Look, no hands! Grid inspection for the future

Powel envisage a future where on-site grid engineers can use voice-controlled apps as their main driver for computer-assisted support during inspections of assets. By freeing the engineer’s hands, a more streamlined process is possible. Additionally, abolishing the need for filling out forms by hand also means that the inspection process will become safer, as engineers are often both hanging several metres above the ground and handling high-voltage equipment.

The event focused on substations as an inspection target. The aim of the hackfest was to have a usable solution, enabling engineers to stay hands-free while performing required duties safely and accurately, by the end of the three days. The solution would be linked to Powel’s Condition Control.

A helping hand

Inspections of grid assets can consist of various tasks. It could for example be to investigate a fault on site according to information received, or it could be to fix a system alarm issue.  Faults are either fixed according to planned maintenance or in the case of serious faults, as soon as possible after being reported. For the purpose of this hackfest, the chatbot was built to support planned- and routine inspections managed in the Condition Control solution.

The idea behind the chatbot is to provide a helping hand to field crew through enabling technology that will be able to provide specific insight related to each particular facility and manage handovers by taking notes of current inspection ensuring no loss of data during inspection. Furthermore, it will support the field worker with other relevant information that might be significant.

Utilising the capabilities of a wearable device, in this case a mobile phone, the user would say the name of the facility being inspected and automatically get the correct checklist.

The chatbot would be triggered at the touch of a button. It would then help the engineer remain hands-free between each task, assisting by noting the task required, answering questions, and providing general information about the situation at hand.

How it works

“A field worker would talk to the application in his or her natural language, which would start the process. Speech recognition transforms voice into text using the Bing Speech API in Microsoft Cognitive Services, returns it to the mobile app, sends it for translation and finally returns it to the app as a fully translated text,” says Simen Karlsen, Smart Grid Enabler in Powel, who continues to explain the steps in the process.  

“The app then continues to pass the translated text to LUIS, a language understanding tool from Microsoft, based on machine learning. LUIS captures the intent and entities and sends the response back to the mobile app. The mobile app sends the detected intent and entities including the field worker ID to the Microsoft Bot Framework, which communicates based on a request to the Powel backend. The appropriate response is then sent back to the mobile app, which translates the text into the field worker’s language and talks back to the worker,” he finishes.

One of the aims for the chatbot was for it to make it possible to connect it to other relevant systems in order to provide the field worker with necessary information. During the hackfest, participants sought to provide a simple chatbot capable of conversing dialogues to external systems and returning valuable insight that could increase efficiency during an inspection. 

"One feature in particular that really extends the usefulness of this bot was the integration of the Microsoft Bot Framework to the Powel backend system. We managed to successfully integrate the bot to the relevant systems that would, for instance, allow field workers to extract valuable information through the use of natural voice such as history of inspection at specific facilities, specific parts prone to constant maintenance or the quickest route to the next critical prioritised facility that needs remediation,” says Karlsen. “Microsoft’s Bot Framework and the Cognitive Services will help us in Powel to empower our customers by making their working day simpler, safer and more efficient.”


From hackfest to reality

The hackfest was an intense but successful experience and the team managed to create a solution built on the latest technologies. As well as developing a working app, the hackfest was a great opportunity for Powel’s developers to familiarise themselves with the Microsoft Bot Framework and CaaP, Conversation as a Platform, technologies. The Proof of Concept (PoC) bot developed during the hackfest will now be used as a foundation for us to move our own bot into production after some further development.

“The cool part about this hackfest is that we managed to build something that actually works and could be put into production in a short timeframe. With LUIS and the Microsoft Bot Framework, we are now able to give the field worker the support he needs when he needs it and without the use of hands. This is a totally new user experience, heavily influenced by Microsoft, an experience we can bring into our customer conversations across the domains within Powel,” says Øystein Askeland, Interaction Designer, Powel.

Powel is dedicated to continuing the development of the chatbot over the next few months to extend the current capabilities to include analytics and historical view of specific parts before going live. We will also look to extend the concept into other business focus areas within Powel.

The solution was built using tools from Microsoft, leveraging Chatbots and CaaP technologies. We created a bot framework in the dev.framework.com space, and also created a Web API Bot framework site in Azure. We then connected these two together. We set up Continuous Integration (CI) in Visual Studio Team Services where the bot would build and deploy to Azure on every commit. We then proceeded to add LUIS integration to start working with the intents. At this stage, 15 intents were implemented.

All code for the hackfest can be found in the following GitHub repository: https://github.com/PowelAS/field-worker-bot