In-Store Proximity Marketing by Means of IoT Devices

. Recently we observe a boom of e-shopping and e-marketing, with plenty of tools, systems, and improvements to encourage customers to purchase more goods and services and to cut off the costs. One may enumerate here the recommendation systems, JIT strategy, instant shopping channels connected to the advertisement, personal targeting, and many more. To facilitate the implementation and operation of such systems, we collect several data related to users' behavior by means of cookies, server logs, link and timing analysis, etc. Up to now, limited attention has been paid to apply all of these mechanisms to real stores and marketplaces. To fill the gap, in this paper we propose a multi-level system to (1) contextually analyze customer behavior at a shopping space by means of Internet-of-Things devices and services, (2) process this information at server-side to compute some instant purchasing recommendations and incentives, and (3) immediately send these recommendations to the customers, either in a form of classical marketing message, or as a personal advice, possibly linked with a discount, “special offer”, etc. In such a way, each customer is served personally and thus has more motivations to buy the recommended or discounted items.


Introduction
Characterized by continuous progress and covering more and more domains and application areas, new communication and information technologies (ICT) changed our life completely. Nobody is now surprised by new proposals to solve everyday problems and to make life easier, just by applying a new device or automated service. Recently we observe this trend also for the activities related to the mass market, from surveys on social opinions and acceptance, via traditional marketing, to recommendation systems. The general trend is to encourage customers to buy more goods and services and to cut off the costs. To facilitate an implementation and operation of such systems, we collect several data related to users' behavior by means of cookies, server logs, link and timing analysis, etc. This trend is especially visible for network-based versions of classical market processes, such as those related to e-shopping and e-marketing. Plenty of ICTbased solutions have been proposed towards modern channels of the advertisement [1], including personal and instant marketing and targeting (nearly approaching famous Minority Report movie [2]), facilitating the shopping process -recommendation systems [3], just-in-time [4] orders and deliveries, and many more.
Up to now, most of the efforts were put on improving the electronic channels and e-* versions of traditional processes. For example, e-shops extensively use recommendation systems based on users' activity logs, personal targeting based on history and recent decisions. E-marketing tools, such as Google AdWords [5], are concentrating on the network and computers/smartphones for searching and accessing the items. However, so far limited attention has been paid to apply all of these mechanisms to real stores and marketplaces. This situation was caused by a limited possibility of gathering the personal information of the same quality at physical places and in a real situation, in comparison with their e-* counterparts. For example, it is quite trivial to analyze server log to filter out some information about previous "shelves" (i.e., Web pages) visited by the user to guess primary interests of this user. However, in a real store, similar information is not so easy to achieve, as the users are not individually tracked across the shopping area. Undoubtedly, one may point out that a customer could be tracked by human personnel, but this approach seems to be much too expensive and as such is usually reserved only for a limited number of stores and situations, such as selling the jewelry or similar luxury items.
With the introduction of the Internet of Things [6] (IoT) and Services (IoS), these limitations seem to be relaxed. Automatic cameras, equipped with face recognition and behavior analysis, may play a role of human personnel and continuously track customers in a shopping space [7]. Intelligent shelves may detect the fact of taking some goods, even temporary, and guess what the client is looking for [8]. Smart shopping carts, counting the items, may recommend buying some more products, supplementing the current shopping [9]. And last but not least, personal smartphones with such utilities as "a list to buy," medical contraindications and recommendations, or even some keywords "not to forget" may facilitate the choice for undecided and dis-oriented customers [10].
To our best knowledge, so far limited efforts have been reported joining the abovedepicted technologies and systems into one consistent proposal, to facilitate shopping in the real stores and marketplaces. Among the most recent proposals, we should point out the work on off-line recommendation system for the real stores and customers based on Oracle tools [11]. However, this one and similar proposals operate on nice diagrams and ideas only, leaving apart the technical solutions and more in-depth research on system requirements and functionality. Also, the proposals are usually addressed to large supermarkets and general stores, bypassing the needs of small stores specialized in certain types of the items such as shoes or clothes. Little work is also done on prerequisites' analysis related to the set of necessary local devices and services, organizational changes, possible profits, etc. Instead, the proposals concentrate on the traditional way of usage of mobile phones -scanning the barcodes or RFID tags and browsing the product ratings and opinions [12], so-called on-shelf-availability ("temporary out-of-stock") feature [13], supply chain management [14,15], etc.
To fill the gap, in this paper we propose a multi-level system to (1) contextually analyze customer behavior at a shopping space by means of Internet-of-Things devices and services, (2) process this information at server-side to compute some instant recommendations and purchasing incentives, and (3) immediately send these recommendations to the customers, either in a form of classical marketing, or as a personal advice, possibly linked with a discount, "special offer", etc. In such a way, each customer is served personally and thus has more motivations to buy the recommended or discounted items.
The recommendation/advertisement is computed in a cloud, thus enabling cooperation of several small marketplaces towards a common sales strategy. Separately, none of the small stores is able to implement and utilize such complicated system. However, all these stores together may elaborate a common solution, based on the past analysis of many customers and their final decisions. For example, one may compute the time a customer is standing in front of a shelf with some items, sending the "special offer" only to the hesitating customers, different strategy may be applied to a person with or without a child, to a person cruising across the shopping area vs. the one concentrating on specific item, color, type, etc.
To facilitate information exchange for local context and customer behavior among several shopping points, we propose to apply so-called virtual IoT devices, thus normalizing the local information to be processed at the server side. To this goal, we propose an ontology of IoT devices, as well as a way to group the devices into conglomerates with enhanced functionality. Normalization makes it also possible to enrich the local context by some external information, in case some necessary local devices are missing. For example, data about air quality, humidity, etc. may be generated for the whole shopping mall, assuming these parameters are similar for all the places and thus easily shared. Similarly, personal information about the customers, generated as a result of observation at given location, may be further used elsewhere.
The remainder of the paper is organized as follows. First, in Chapter 2 we shortly describe current state-of-the-art for instant marketing in real stores by means of ICT. Chapter 3 is devoted to a proposal of a new system architecture aimed in collecting the personal information at the place, standardizing it and enhancing by a local context, and sending to the cloud to compute personal shopping incentives at real-time. In Chapter 4 we discuss possible ways of standardization and virtualization of local IoT/IoS utilities, followed by a discussion of remote server-side functionality. Finally, Chapter 5 summarizes the proposal and depicts the future work.

Current Usage of ICT in Real Stores and e-Shops
At the moment, the usage of ICT technologies for shopping and marketing may be divided into two disjointed areas: those used in real stores, and those specific for eshopping. The first area is usually related to the following proposals: loyalty cards, alert systems and security services, customer-flow analysis and tracking, RFID for supply chain management, and, as for the scientific proposals rather than the reality: "intelligent" shopping carts and shelves, and personal advertising and filtering.
Loyalty cards. A loyalty card is a discount/payment card offered to a customer. With such card, a customer may count on lowering a price for "special offers", cyclic and single discounts, exchanging collected "points" to "gifts" or price reduction, etc. Instead, the customer is obliged to register their shopping details, such as timings, prices, choices, etc. Such information is then used to improve the organization of the store, to determine rates and discounts, to assess the marketing effects, etc.
Alert and security systems. Several technologies are used against crime and theft, to mention magnetic and radio tags for expensive items, security gates, monitoring systems based on cameras and detailed inspection, etc. Most of these techniques require direct human control. Thus they are usually restricted in usage by delays and the law (e.g., in order to catch a thief one must call the police forces).
Customer-behavior analysis and tracking. Tracking systems, based on automatization of monitoring of some characteristic features of the customers, such as positions, movements, stops, delays while making a choice, etc., are usually based on cameras and video analysis. As it is almost impossible to track each individual customer separately, except for very artificial conditions (such as a single customer for the whole store), usually the videos are analyzed only statistically, e.g., to point out a place frequently visited by the customers to establish a marketing point, to detect if the organization of the shelves is correct, etc.
RFID. Radio-operated tags are used mainly to facilitate supply-chain management [16]. Due to several restrictions of the technology, related mostly to the physics of radio-signal propagation, it is usually not possible to track individual goods unless they are big enough or separated of each-other [17]. As a consequence, RFID tags are mainly used for large containers and packages, and for the shelves rather than individual items and goods. As for the latter, traditional bar codes are applied, however, with limited tracking possibilities due to the need of line-of-sight (LOS) inspection [18]. Even if there are some attempts to track RFID tags of individual items, not only inside the store but also after shopping (to mention the analysis of wastebaskets near homes to get the popularity of a product at given area), this technology seems to be quite specialized and hardly applied for direct and personalized marketing.
"Intelligent" shopping cart. Shopping carts are used in larger stores to temporary collect the items to be later paid in total at the cashing point. There were some proposals to include a display at the top of a cart to enumerate all the items and to count the total price for the contents of the cart. Such carts were tested by several market leaders, however, never gaining significant attention, for several reasons. First, due to technology restrictions, it is hard to detect the contents of a cart automatically -either a customer is forced to scan a barcode/RFID tag manually, or some tags are omitted because of screening of radio signals or above-mentioned LOS restriction. Second, as the customer was aware of the price at any moment, frequently they decided to remove too expensive items from the cart. Also, the accumulators for the displays to operate were heavy and needed frequent recharging. Recently, there were some attempts to replace the displays by personal smartphones; however, this approach was hardly accepted by the public mainly due to the concerns on privacy.
On-site personal recommendations. Similar to advice provided by a human assistant from store's staff, one may imagine similar assistance given by computers, preferably personal smartphones. An imagination of how it may look like is given in famous "Sixth Sense" presentation at one of the TED conferences [19]. For example, a customer is scanning a tag or taking a photo of an item, and after a short time, they are notified with a red or green light if the product is suitable to buy or not, due to customers' opinions, medical recommendations, or simply a previous choice and individual/family experience. Similarly, the smartphone may guide the customer to complete the list of items to buy, navigating across the shopping area. However, as it may be seen for the "Sixth Sense" project, such assistance is very restricted according to the place and situation, and, without the support of complex external systems and services computing the recommendations, seems to be impractical.
Note that none of the above-mentioned techniques is able to detect and analyze personal information about a given customer at real-time, i.e., at the moment of choosing particular items by this customer. All these techniques are based on collecting some information and processing this information off-line, mostly for statistical purposes. Such experiments as "intelligent shopping carts" and "Sixth Sense" project are very interesting, however, quite restricted and impractical.
On the other hand, for e-shops one may enumerate several online technologies, to be used to assist the customer in taking the purchasing decisions at (near) real-time. These techniques include log analysis, personal targeting and recommendations, instant, and individual offers both for the items and their pricing.
Log analysis. Log analysis is a very useful tool for gathering the information what a user was doing while visiting a website, which pages they saw, for how long, how they navigated, etc. In general, analyzing this information statistically may enable targeting a specific user, i.e., linking this user with a well-defined group of "similar" users. Then it is possible to predict the future customer activities, and even to facilitate some desired activities by below-discussed personal marking and recommendations.
Personal targeting and instant offer. A customer, identified by a statistical member of a group of "similar" customers, may be treated in the way related to the group. For example, once a customer watched some action movies only, it is quite probable that they will get another action movie. And if they are about to leave the system or hesitating for a longer time, it is valuable to offer them a personal discount, e.g., 5%, for the next action movie to be realized within a few minutes.
On-line personal recommendations. Everybody knows a bit of advice while buying some items in an e-shop: "customers who bought this item also bought items X, Y, and Z." The choice of such recommended items is based on the statistical analysis of shopping of the past customers [4,20]. For some applications areas, this simple approach is quite valuable, such as for computer games, electronics, furniture, etc. Recently, a specialization of this approach for mobile devices and services took attention of the researchers -the specifics for the mobile world is a base of huge expectations towards the automatization of this market.
Collaboration of several stores at the base of a common customer. The abovedescribed personal recommendations are not necessarily addressed to a single store and/or type of an item to be purchased. Instead, it would be much more useful to determine a global profile of a customer, to be used for several item types and store locations. To this goal, cooperation is needed of several stores, based on the storeindependent identification of the customer. The latter might be implemented as a common loyalty card/application, e.g., the one issues by a shopping mall. Note that such identification is not obligatorily linked to processing personal data of the customer. Instead, it is enough to generate a unique number being a representative of the customer -this task may be performed as a part of the subscription procedure. It is, however, an open question how to determine the set of marketing rules to cover different types and domains at once (such as "a customer who bought red shoes will be probably interested in red blouses").
Note that, on the contrary to the real-store technologies, e-shopping techniques are based on online analysis and instant recommendations. However, they are built on the information that is not available for automatic treatment at a real place, such as timings and presence of given person at given place and time, careful inspection of the items before shopping, returning to the place/item after some time, choosing a product from a group of similar ones, hesitating before final decision, etc. The question is if we really cannot get such information in real time and place, to shift the profits of e-shopping analysis to the area of real stores and marketplaces. We think that gathering such information is now possible with the recent advances in the field of Internet of Things and Services. Our idea is described in the next chapters.

Proposed System Architecture
The proposal is based on two fundamental assumptions, these are the following: • there is a local system of IoT devices (both real and virtual) for each place, such as a store or a market/exhibition place; in general, IoT devices gather some information about local context (HVAC -heating, ventilating, and air-climatization conditions, positions and movements of customers and visitors of the store, characteristic features of humans and their activities -staying, fast/slow moving, watching, listening, etc.); • the local contextual information is sent to the cloud to be processed at server-side and to compute some shopping incentives, which in turn is sent back to be presented locally to customers. As a result, we obtain a solution based on the cooperation among several, possibly small or medium-sized stores, to attract the customers with JIT marketing information, incentives, and offers. Locally gathered information about the context and customer activities is processed in a centralized way, thus making it possible to share the costs among all the business partners involved. As the cost of local IoT devices is reasonably small (c.f., further discussion on a set of possible hardware/software to be used for monitoring the customers), and the cost of quite complicated server-side analysis of the data gathered locally is shared, from the point of view of a single shopping point the whole system is effective and not very expensive.
In a natural way, the system is divided into two primary layers: • local IoT network, responsible for collecting the information of a different kind, nature, and meaning. To this goal, we propose to use: (1) smartphones with a loyalty application, working according to BYOD (bring you own device) idea and acting as a broker between the system and a customer, (2) geolocation beacons, to identify a place inside a store, such as a single shelf or a stand, (3) Near-Field-Communication (NFC), Radio-Frequency-Identification (RFID) and QR-code tags, to enable information exchange at the distance of centimeters, for example to realize the payments or identify given item, (4) local WiFi or global GSM-related network (such as LTE) to contact the cloud and obtain marketing information, and (5) local sensors of different purpose, to collect the data of the local context (such as HVAC conditions, but also visual and acoustic background, weather forecast, image analysis towards the detection of presence and activities of humans, etc.). All this information is multiplexed by specialized hardware (based on Raspberry Pi and AVR processors/modules), standardized according to specific ontology and sent at the request to the cloud, to form a local context. • remote server operating in the cloud, to process the context to compute JIT advertising, to be sent back to the customers to their BYOD devices. The server is using several artificial-intelligence tools, such as neural networks and reasoning systems, to (1) possibly automatically extract the important information from the local context, and (2) process a set of predefined (expert-generated) and dynamic rules to choose the most appropriate marketing information or a personalized offer, to be presented to the customer. The nodes exchange the information in a standardized way. To this goal, we plan to prepare an extendable ontology of virtual IoT devices. Each virtual device is based on an arbitrary chosen set of physical IoT devices operating locally and remotely, and is characterized not only by some information collected (such as a temperature, air pressure, a number of customers in a store, their positions and distance from a shelf, timings related to the customers and choosing/regarding the items, etc.), but also by the accuracy of the data. The latter parameter is used to express the trust and the importance of the sensor. This approach makes it possible to report some external sensors/information as a part of the local context, however, with limited accuracy. For example, there may be a thermometer reporting the temperature close to a shelf -the accuracy is then represented as 100% maximum value. If, however, the store is equipped with only a single global thermometer, such centrally measured temperature is accompanied by the accuracy limited to 70%. Further, if the store has no own thermometer, but there is a global temperature sensor for the whole shopping mall, the information from such global sensor is reported with the accuracy at the level of 50%. Moreover, the accuracy may also be used to share some information among several locations. For example, it may be assumed that the temperature in each store in the same shopping mall is similar -then one spot may get the data from the neighbor, with the accuracy at the level of, e.g., 60%. Anyway, the information from the virtual sensor is always available, differing only in the accuracy and trust. This idea is applied to all the virtual sensors, shared by the stores if needed and always reported in a standardized way to the remote server.
One should note that the presented approach preserves the anonymity and the privacy of the customer. The system knowns and is able to identify only the smartphone, more precisely, the loyalty application, by its registration (serial) number. Name and surname, address, e-mail, any other identification information -none of such information is needed by the system to operate. In the matter of fact, the system tracks past and current activities of the customer to better adjust marketing information and personal offers, and send appropriate messages in real-time during shopping activities. But this is exactly the way human shopping assistants act -they identify the customers visually, they come closer and offer their help to facilitate the choice. So, one may expect that the level of acceptance of the automated assistance will be similar as for nowadays human staff.
Note also that the shared database of the customers is a base for efficient collaboration of several stores towards a common customer. While concentrating the knowledge on customer's habits and choices at a single place, we obtain a possibility to apply this knowledge at many places and coincidental interactions. This way of the cooperation is especially useful for small and medium enterprises such as the small shops in a shopping mall, as typically the customers are going to visit a store by coincidence, from time to time, also searching for something not usual and planned.

Typical Scenario of System Usage
A store is equipped with several physical sensors: HVAC conditions, RFID/NFC sensors to localize the items on shelves, devices registering an interaction of a customer with an item (such as getting an item from the shelf, unpacking it, testing, verifying the size in a fitting room, comparing fashion, colors, etc.), camera-based analyzers (person locators, crowd detectors, localizers, movement detectors, timers, size/position estimators, such as cameras able to distinguish a mother and a child based on the growth, etc.), and similar sensors. Some information may be also provided manually by shopping assistants, such as the mood of a client, ambiance, impression, etc. All this information is standardized and collected by a local multiplexer.
At the moment, a customer is approaching a place marked by a geolocation beacon -a stand, a shelf, or, in the simplest case, main entrance to the store. Once fetched by the customer's smartphone, beacon signal pops up a loyalty application. The application contacts the server in a cloud and reports beacon identifier. The server localizes the store and detailed place and asks local multiplexer of this store to send all the information regarding the virtual sensors. Then, the data is processed to compute a JIT advertising or personal offer and further sent back to the customers.
Note that the virtual sensors are chosen based on customer's location, but they are not necessarily operating precisely at this location. Due to the standardization and virtualization of the sensors, the server may operate in the same way for any other place, taking only into consideration, if needed, the accuracy for the sensors reported at this location. Note also that the smartphone itself may extend the context, for example, to report favorite color, size, fashion, etc., of the "most wanted" item. This information may also be sent to the store staff, to better assist the customer at the moment. For example, by scanning a beacon installed close to an entrance to the shopping mall at an exhibition place with some mannequins and clothes, a customer will be given by a map how to get to the store, and inside the store a ready-to-test item will be waiting for them, just prepared by the store staff, fitting customer's size and favorite color. This is also a good point to start cooperation among several stores. For example, one may point out a case of shopping the trekking shoes at one location and providing there some more information to complete the rest of the equipment in the other stores near-by.
It is an open question how to generate the marketing information in JIT manner on the server side. At the very beginning, the system will be based on a set of fixed marketing rules, prepared by the human experts and parameterized by the information coming from virtual sensors. For example, if a customer is standing in front of a shelf with some clothes for more than 30 seconds, hesitating what to buy, the store may offer them a 10% discount on condition they will complete the choice and report to the cash register in the next five minutes. The discount will be automatically deducted from the price during the payment via the smartphone and further registering this fact by the loyalty application. Such the set of fixed rules may be also improved by a dynamic analysis realized with some artificial-intelligence tools, such as neural networks or extendable expert systems. However, this aspect of system functionality needs further research, and probably such improvements will be introduced step by step after some experiments with real customers and situations.

Conclusions and Future Work
The proposal depicted in this paper is a new approach to generate marketing information and purchasing incentives in real-time and for real locations (stores, exhibitions, marketplaces, etc.). The advertisement will be selected based on the sensors located near the place where the customer is currently located, their behavior and current activity, appearance (e.g., an adult or a child), and the general context. The location of the customer will be determined by their personal device after detecting a geolocation beacon or after scanning an NFC / RFID / QR code. The location will be used by a remote server to determine the store and possibly also a detailed place inside this store. Then, a set of IoT devices will be contacted to provide the information about the local context, describing not only traditional parameters such as the temperature, air pressure and flow, noise, etc., but also customer activities in a shopping area such as positions, movements, gestures, actions (e.g., getting an item from a shelf, unpacking it, walking with it to the fitting room, etc.). The contextual info will be sent to the cloud in a standardized way. Based on this information, an advertisement or a personalized offer will be generated and sent back to the customer.
The system is a chance for small and medium-size stores to cooperate to attract the customers, with the promising technology and the costs shared by all the business partners. The proposal is an adaptation of existing tools related to digital marketing to the level of the real store and a specific purchasing situation. A customer subjected to such a marketing stream will probably treat and feel it in the same way as assistance provided by the human staff -one should expect a high degree of acceptance of the marketing messages and shopping incentives and an increase in customer satisfaction with the service.
There are several aspects of the proposal not covered by this paper, mainly due to the lack of space. One should mention here security and privacy issues, anonymity, a possibility of abusing detailed information about a customer for non-local purposes, how to encourage and reward the customers for entering to the system, and many more. All these aspects should be discussed, also addressing the recent European regulations on data privacy. This is our general plan for the future work. Also, we would like to study the economic base for the cooperation of several stores, taking into account the fact that some of them are possible competitors on the same market (area, shopping mall, city, etc.). Regarding the above-mentioned anonymity, some would claim that in case of a single store anonymous access is possible; however, when dealing with cooperation of several stores and a common customer, the customers are anyway no more anonymous as they have to register for the system. This claim is only partially valid. As it was pointed out earlier in the text, there is no need to identify the customers by their personal data. Instead, a serial number of the loyalty application may be used to this goal. Thus, as a decision to keep the application or uninstall (reinstall) it is left to the customer, we fully conform to the accordance of "a right to forget" and similar rules.