General Introduction

The famous Silicon Valley investor Marc Andreessen said, “software is eating the world”^[1]. By, which he means that increasingly we are bringing software into core of systems to increase efficiency, lower down the cost or time involved in the functioning of a device and its processes.

As software proliferates into various walks of life, homes and the home equipment are not left untouched. As the software touches more and more everyday objects we see in our houses, it is creating a big market opportunity world over. Various estimates of the market size of smart home put it around USD 200 billion by 2025^[2].

The ‘smart’ in smart home is coming through the software component or that software component further converted to specialized hardware ( system on a chip) , one of the core components that we explore in this chapter is the ‘smart’ aspect of these devices. In later sections of this chapter, we refer to these connected smart devices as IoT, Internet of Things. A layered approach of understanding the IoT devices is given below.

Hardware Component of the Device

The hardware aspect of a smart device would consist of the core component of the device. By ‘core component’ we mean the traditional hardware of a home appliance whether it a refrigerator, air conditioner, fan, or a light bulb to achieve its primary objective of cooling, air flow or light. Additionally, it will have sensors, which provide the input from the environment. Like an air conditioner may have a thermostat to obtain the temperature of the room, the bulb may have a motion sensor, so and so forth. In the world of smart devices sensors play a key role in enabling innovative usecases of convenience, energy efficiency, security, health and wellness or any other important aspects.

We believe the next wave of innovation would enable newer kind of sensing (new sensors), making existing sensors cheaper and enabling data privacy in the sensors at the edge.

In order to make sensing cheaper in some domains, we may find interesting combination of cheaper sensors using edge-based AI, trained for sensing a specific use case[1]. For example, microphones are very cheap and if they can be combined with edge AI, can detect variety of sounds that may indicate running of a fan, air conditioner, water tap, or even a fall/object drop etc. We have seen privacy research on sensor data where amount of carbon di oxide in a room can reveal if people are sleeping, smoke in kitchen can indicate eating dinner at home and so and so forth[2][3]. So, the direction of newer research will try to make AI enabled edge devices to share the data selectively, for example a CCTV might blur the face and body parts of a person member if there is nudity in the scene[4].

We further need two more hardware components that of computing and radio interface for connectivity. So, the device may have a Bluetooth or a Wi-Fi interface and the computing component which can run the connectivity protocol using the radio protocols such as a ZigBee or Wi-Fi.

When equipment manufacturers chose the sensors, computing and connectivity technologies for a smart device. A big consideration for them is cost or put the other way the more hardware a smart device packs the more costlier it is. Hence, majority of the time the hardware for computing and connectivity takes a backseat in terms of capability and cost and pushes the burden of computation and storage to cloud component of the IoT Stack.

Firmware enabling Communication in IoT

The communication between the humans and the devices happens through various interfaces for the purpose of information and control. Some estimates suggests that there are 8 networked devices per person in a home in US and it will grow approximately to 13 per person in the next two years^[1].

If we take an average family size as 4 people, two adults and two children, they coordinate among each other using a human language/gesture in person or the interaction mediated through technology, primarily mobile messenger applications like WhatsApp etc.

If you add to this the 8 devices per person, the complexity of permutations and combinations in the same family grows multifold. Theoretically speaking in a home of 36 entities including humans and devices there are more than 1000 ways they can interact with each other. If we add to this the context situation then the trigger of communication can be even millions of possibilities. Navigating this complex web of possibilities with the modern offerings of artificial intelligence and similar advances can enable the human experience to a level, which is no less than magic!

Let us further break down this communication to the dimensions of content and medium. We will delve deeper into the content dimension of this communication in the hubs section below. In this section we will deal only with the medium dimension of the communication.

For the human-to-human communication sans technology, it used (gestures or language) visual and audio communication. Hence, the medium is in the visual light and audio spectrum. For devices on the other hand the medium is radio waves which fall in various frequency ranges not just visual light range. We will refer to the various technology solutions that are commonly used in the IoT devices are wireless technologies.

The various wireless technologies that are prevalent in IoT devices are using different frequency ranges and an implication of that they have a different data rate, range, power and cost. This variation allows them to be used for different use cases. For example a smart meter at home does not need a very high data rate but a longer range and ultra low power. Hence, a LoRa based solution might be a good choice. On the other hand, a smart bulb in a house might not be deprived of power source, might have a shorter range and a low data rate, so it might come with any of the wireless technologies like bluetooth, wifi or zigbee. Below are some parameters/constraints on basis which a wireless technology might be chosen to be part of an IoT device.

Lifi: A new emerging wireless technology that can seamlessly integrate into our homes

While each of the above wireless technology candidates like wifi, zigbee or LoRa deserve an introduction. We will elaborate on this section of a candidate technology which is called Light fidelity, or in short Lifi. The term was introduced by Prof Harald Hass first in 2011 based on his research on using visual light for communication[1].

Light bulbs are already common technology in households for more than hundred years now and the recent upgrade in technology of light emitting diodes (LEDs) make them as a building block for enabling communication apart from their core function of producing light. Since visual light is also part of the radio spectrum it can also carry information when modulated over frequency much like the other radio waves. Since, the visual light is of high frequency the data capacity it offers is also huge. Apart from building on an existing omnipresent building block of LEDs and high data speeds it offers also two more advantages. Since, there are many lighting fixtures indoors, a Lifi unit can also be utilised for indoor positioning[2]. We all have some point experienced a scenario where a neighbour trying to snoop on our home wifi password, the 2.4 Ghz spectrum which wifi uses can travel beyond the walls. Lifi on other hand based on visual light remains confined into the four walls. Hence any applications of using a radio communication where line of sight is available for use and yet outsiders can be prevented as the waves do not cross the wall of the house, Lifi makes a good candidate for that scenario.

Given these properties, Lifi is also being considered as a candidate technology for last mile 5G technology[3]. We hope that in near future you maybe consuming Lifi based movie streaming on your phones at home but maybe in also airlines where they prefer customers to bring their own devices to watch content provided by the airlines[4].

Wired Communication

We are not covering in this chapter in detail the various ways wired communication can happen in a smart home. As we consider that sooner or later most communication among the smart devices will happen over a wireless technology. However, for the purpose of completeness we list various types of wired communication that we observe in the home setting.

Dedicated Channel: Having a fixed wire to communicate, such as a landline phone at home or an analog speaker.
Power over ethernet: Distributing internet connection within a house over the existing wiring for power within a house.
Powerline communication: Using existing power distribution lines for sending communication. This of course requires at the both ends a converting unit which can separate the communication signal from the power line.

Interfaces: their role and evolution

The traditional interfaces of devices at home have been manual, a physical power plug or a button in the device such as a washing machine or a fan is all familiar to us. As the physical buttons have gone now from the mobile phones in the last 14 years or so^[1]. The physical buttons from many other devices are also starting to virtualize. Increasingly the devices we buy have a screen in it or can be connected to the screen in our hands i.e our smart phone.

Purpose of the interfaces

The interfaces (application on our mobile phone) for devices can be used in three ways:

Primarily to operate them, like switching on and off a device. For example: you are lazy and you don’t want to stand up to shut down the air conditioner, the remote control of the air conditioner is also lying in another room. What if you could use your mobile phone to do the same?
The second to build intelligence and convenience in the operation of these devices. Example: Switching on the air conditioner fifteen minutes in advance of your estimated time of arrival at home. For this use case to work somehow the Global Positioning System of your car or mobile phone needs to be connected with the interface that controls the air conditioner. Increasingly people want to automate the routine aspects of their life and this is where the intelligent rules can be defined and invoked on the basis of contextual / pre-determined information. Hence, you need an interface to import/create such rules and set your preferences.
Information as a means to persuade and influence the behavior: The social media and other behavior change technologies have demonstrated that you can prime the behavior of users by showing information to them. This could be manipulative in certain context and also to induce good behavior in other aspects[2]. This aspect of information sharing with a purpose can be very impactful in energy and cost saving dimensions.

Ways to interact with the interfaces

As hinted above that initially we interfaced with (dumb) devices through manual, push button approach which was also in some usecases were sensor information driven, like proximity or motion for example. As the interfaces got virtualized we now increasingly use them through our mobile phones and also able to build additional intelligence which is not resident on the device or its connected sensors.

Already now for several years there is an increasing use of smart speakers at home, where computing enabled speakers (with in built microphones) operate on human voice and command. Hence, the user interface is soon becoming invisible and technology is seamlessly trying to retrofit in our everyday life. People can now have these smart speakers in their bathrooms and ask them to play music of choice, read out stock market information to list appointments for the day, as in that setting mobile phone might not have provided a hands free approach to information as the smart speakers can.

We may say the current era of smart devices maximize convenience.

There are numerous companies right now working on brain computer interfaces[1], hence in few years we will also be able to access our devices using thoughts. It might not be a distant future where your house can sense our mood and adjust light or music to make us feel better.

The big question is, as a human species our brain has tricked our body for eons and make us do things to seek pleasure. Obesity or gambling might be the manifestations of the manipulation of our brain. If we could trick our brain for good, could we improve our lifestyle? Many people already blame companies like Facebook in manipulating ‘brains’ and to addict users to increase the number of clicks[2].

The interesting question in future would be, when visiting a friend’s house, would you be comfortable using brain computer interface, fearing you are not manipulated and an unwanted thoughtware not installed in your brain! We may think of ethical and societal implications of giving access to our innermost thoughts to machines! The 2013 movie ‘her’ showcases the imagination of a filmmaker on how seductive AI enabled interfaces maybe[3].

The future smart devices may maximize happiness on the other hand!

Cloud Computing

Cloud computing has seen an exponential growth in the last ten years and it is primarily because of the operational efficiency that it brings at a scale. The agility that cloud offers for a business is huge and can enable optimizations, data analytics, magical appearance of new features overnight and enable API and Platform economy at scale. In the smart home context, the word ‘smart’ increasingly relies on intelligence derived out of data. The multipurpose computing hardware which is packaged and shipped to us in our IoT devices is not designed to compute and store huge amounts of data. Hence having huge amounts of horse power over the Internet enables the small device, to outsource heavy calculations, storage of data and intelligence that can be derived from that data to cloud computing. This data collected over the cloud is also useful for the service designer to share analytics with the users to influence and inform them.

Let us take example of a small camera that maybe installed in the house for security purposes at the door. The camera in itself cannot do much as it is a dumb device and just captures the frames. When that camera can connect to cloud, if it has a connectivity module in it, it can send all the captured frames to the cloud. The storage on cloud keeps getting cheaper every year with the falling price of storage, hence the consumer who avails the cloud subscription can now access this video feed in real time or otherwise using their mobile devices. Which gives them the independence from the constraints of time and space. The camera company gets to upsell the storage and access service to its consumers for an additional charge.

Sifting through the hours of camera feed is not an easy job. Hence, if you are a worried home owner when away from a locked home, you might have to be hooked on to your camera screen and stay anxious all the time. On the other hand the cheap and scalable compute offered by Cloud enables the service provider to provide another set of premium features to consumers wherein they can get alerts when someone is at the door and it is possible for the camera to learn faces and even send people specific alerts. So, you can know that kids are at home or your spouse is back from office without being hooked on to the video stream all the time.

As we experience the increasing foot print of our digital lives at home with movie streaming services or gaming consoles apart from the work from home culture enhanced by the COVID-19 scenario. We are now increasing relying on cloud enabled services in our domestic lives and indeed it is an integral part of our lives. As these technology trends grow, so is our reliance on cloud computing.

5G networks and optical fiber enabled backhaul, low latency and higher speeds will support a future in our homes where the virtual and physical reality will start merging seamlessly. As dwellers would be able to download virtual reality avatars of their favorite celebrities, sports heroes and engage with them in learning, playing and entertainment. Do family video calls in 3D holograms and much more computing and bandwidth intensive workflows which only few can afford today.

Intelligence from the network of connected devices

We would like to use a loose analogy from human civilization / Internet in this section. When humans connect with each other, they exchange ideas and create literature, art and invent things. We see that when the same human beings connect over the Internet, in the last three decades the creative output has only gone manifolds and large scale problem solving has been possible. A pioneer in the direction of human computation enabled by Internet is Prof Luis Von Ahn. He has created multiple inventions that have enabled the large scale participation of humans to solve problems. CAPTCHA[1], ReCAPTCHA[2] and Duolingo[3] are some of his creations. He says,

“Before the Internet, coordinating more than 100,000 people, let alone paying them, was essentially impossible. But now with the Internet, I’ve just shown you a project where we’ve gotten 750 million people to help us digitize human knowledge.” – Luis Von Ahn

On the other hand the latest key idea in the field Artificial Intelligence is Connectionism, which has given birth to Deep Learning. Quoting from the book of Ian Goodfellow, “The central idea in connectionism is that a large number of simple computational units can achieve intelligent behavior when networked together. This insight applies equally to neurons in biological nervous systems and to hidden units in computational models.” We see a parallel of connectionism in smart devices network as well and believe greater intelligence may be an emergent phenomenon!

Of course smart devices are not conscious or not even close to artificial general intelligence yet. So, this analogy may not closely fit when you connect smart devices to each other mediated by a network. We believe we are yet to see that phenomenon on a large scale and can imagine a scenario where intelligence will not just be at the edges (read smart devices) but may also emerge in the network. For example, if we have to enforce some constraints for energy or creativity on a city or a residential community.

The city mayor may announce a competition for neighbourhoods to emerge as using least amount of energy and network may enable them to cooperate and compete or on the earth day once a year they all might want to collectively switch off their equipment and donate energy for a good cause[4]. The contact tracing based approach of containing COVID-19 cases by countries like South Korea and Singapore are examples of such systematic intelligence that builds on individual information when aggregated.

[1] https://en.wikipedia.org/wiki/CAPTCHA

[2] https://en.wikipedia.org/wiki/ReCAPTCHA

[3] https://en.wikipedia.org/wiki/Duolingo

[4] https://research.aalto.fi/en/publications/evolution-of-user-driven-innovation

[1] https://www.forbes.com/sites/cathyhackl/2020/06/21/meet-10-companies-working-on-reading-your-thoughts-and-even-those-of-your-pets/#3579aebb427c

[2] https://www.theguardian.com/technology/2018/mar/04/has-dopamine-got-us-hooked-on-tech-facebook-apps-addiction

[3] https://en.wikipedia.org/wiki/Her_(film)

^[1] https://www.history.com/this-day-in-history/steve-jobs-debuts-the-iphone

[2] https://www.interaction-design.org/literature/topics/persuasive-design

[1] https://en.wikipedia.org/wiki/Li-Fi

[2] https://arxiv.org/abs/1602.07399

[3] https://www.computerweekly.com/news/252469006/5G-testbed-brings-Li-Fi-broadband-to-Orkney