[Hackathon] Introduction to Easy Medicine App – Team Healthy Healthier

Nowadays, many app has been created to make our daily tasks become simpler and more efficient. One of the most competitive features among apps is automation. That is, humans will only do the minimum of operations and leave the rest to the application. Since Artificial Intelligence – AI has become more and more popular, automation is now everywhere, and at the same time, it has also opened up a lot of possibilities. Easy Medicine is also an application built on AI technology and it helps to automate medication management.

Table of contents:

1. What problems does EasyMedicine solve?
2. How does AI technology participate in our app?
3. Google Cloud Vision introduction
4. AWS Comprehend Medical introduction
5. App’s system design
6. Demo Easy Medicine
7. Future features and functions
8. App limitations
9. Summary
10. Links & references

#1. What problems does EasyMedicine solve?

If you are a healthy person, you might wonder why would you ever need a medication management app. But if you put yourself in the situation of being a person who suffering from many different diseases, then you will see it differently. Even if you can control your schedules well and always take your medicine on time, isn’t it very easy to make you feel annoying that you have to remember all those schedules. Meanwhile, everything can be completely controlled with an application. And you can now focus on things that are more important. There is also a situation that highlights the problem clearly, and that is someone who is taking care of many sick people – a mother, for example. Also, we will sometimes forget to take our medicine, especially when the disease is in remission, we will become lazy. And as a result, it negatively affects our health.

More specifically, we need a tool to help manage the following:

• • What time should you take your medicine today, what are drugs, dosages?
• • To avoid forgetting to take the medicine, we need reminders alarm.
• • What medicines did you take or forget to take today?
• • How long do you need to take the medicine, how many more pills?
• • Store prescription information for easy lookup and reuse.

And the above are the core ideas that form the Easy Medicine app.

#2. How does AI technology participate in our app?

As analyzed in the above section, you now understand why we need an application to help managing medicines list and reminding to take medicines on time.

To make the app functions as easy to use as possible. We need the processes to be automated. Creating a list of drugs must be done with in one operation from the user.

Specifically, the user will take a picture of her/his prescription, then the prescription will be scanned and a list of medicines to take will be generated (with specific dosage information). At the same time, the app will automatically schedule the alarms for taking medicines.

Thus, everything will be completed just by taking a photo. And AI technologies will make it possible:

• • Computer Vision: An AI technology that allows to distinguish objects in a frame. In this case, we need to extract the raw text of the prescription in the image that the user has taken. The Google Cloud Vision API is the tool that will be used.
• • Natural Language Processing (NLP): A branch of artificial intelligence within computer science that focuses on helping computers to understand the way that humans write and speak. This is a difficult task because it involves a lot of unstructured data. Since the content will be prescriptions, the selected AI needs to be trained using medical data. AWS Comprehend Medical is a suitable service in this case.

#3. Google Cloud Vision API introduction:

An API provided by Google. Its advantages come from high processing speed and accuracy, ease of use and has been trained with a huge amount of image data. The extracting text (Detect text) feature from images can handle both typed and handwritten text, also, it works well with many different languages.

Service regions:

• • us: USA country only
• • eu: The European Union

Service pricing (Pricing – Cloud Vision API):

• • Google will charge a monthly fee based on the number of images processed. Each image will be counted as 1 unit.
• • They charge you differently depends which tiers you’re in. For the EasyMedicine app we stay below 5,000,000 units/month. At this tier, it will cost 1.5 USD per 1000 units.
• • For example, every month the application processes 10000 images, then the total amount will be 15USD.

With a fee of 1.5 USD per 1000 units, it is quite expensive. Hopefully, in the future we will be able to find a service with a better price, maybe the service is located in another region and the price will be cheaper.

#4. AWS Comprehend Medical introduction

As an AI service of AWS (Amazon Web Service) NLP (Natural Language Processing), this API takes text as input and returns the extracted medical related information. Because it has been trained with a large amount of medical data, the accuracy of the results is quite high – a very important factor in this application because it directly affects the user’s health. Because it is an AWS service, there are many regions where the service is located. Therefore, the response speed is also very optimal.

Although the information that this AI is able to recognize is quite diverse, we only need to focus on information such as:

• • Drug information: drug name, drug dosage.
• • Time information.

Service maintenance cost (Amazon Comprehend – Pricing):

• • Every 100 characters = 1 unit. There are different tiers, but EasyMedicine will stay below 10 million units.
• • At 10 million units tier, the fee per unit is $0.0001USD. For example, if you need to process 10,000 prescriptions per month, and each prescription will have an average of 500 characters, the fee will be 10000*5*0.0001  = 5 USD.

This is a pretty optimal price.

#5. App’s system design

Below is the system design diagram of Easy Medicine:

To better understand the above diagram, let’s see the application flow: First, when the user inputs a picture of a prescription taken/uploaded, this image will be sent to the server. The server will now call the Google Vision API by sending the base64 encode of the uploaded image. Google will process and return the entire text content extracted from the image. For instance, this prescription:

The server then uses the scanned text to call the AWS Comprehend Medical API to request medical data extraction. Here is part of the returned result:

Comparing the JSON result to the original input image. We can see how the AI uses the predefined structures for the information extracted from the prescription. With this JSON data, the application then re-process it to generate the medicines list. Specifically, we will focus on handling the following attributes:

• • Type is BRAND_NAME orGENERIC_NAME:  this is the medicine name.
• • Type is  DOSAGE or STRENGTH: this is the medicine dosage.
• • Type is FREQUENCY: this data is crucial for creating a schedule to take medicine.

From the extracted information, we need to process that information to create a list of medicine and a schedule:

Medicine name: It’s easiest to obtain because it is not abstract data. As long as the data exists in the AWS database there wouldn’t be any problem.

Dosage: Because there will be more than one DOSAGE attributes returned. We need to distinguish what is the actual dose of the drug, and what is the amount to take at a time.

• The medicine dosage will usually come in units of mass/volume of the drug. Such as 100mg or 400ml. Using the following Regex (Regular Expression), we can easily extract the drug dosage: /([0-9]+)( ?)(mg|ml)/

• The amount to take at a time can be in the form of a mass (50mg) or it can also be in the form of words (1 tablet, one vial, …). The method to extract this data will also be the same as above.

Medication schedule:
• Time to take medicine in the form of pharmacy abbreviations code: for example, a presence of q.i.d on the prescription is equivalent to Four times a day. This means that the application will schedule alarms to take medicine 4 times a day. In the morning (7:00 AM), lunch (13:00 PM), afternoon (18:00 PM), evening (21:00 PM).

• Time to take medicine in the form of timeline: The application needs to schedule a medication appointment to match the time described. This form will have a diversely of cases. For example Before breakfast (every morning at 7:00 AM) or Twice a day (morning – 7:00 AM, noon 13:00 PM). To be able to convert from timeline to a specific schedule. We can use Regex, such as:
/(every|each|after|before) (breakfast|lunch|dinner|meal)/

The final step is to schedule a time to take the medication. Based on the created list, the system will send push notification to the user’s phone. This allows them to take their medicine on time and with the correct amount.

#6. Demo Easy Medicine

Home Screen:

• • The main feature of this screen is to control your medication schedule.
• • It’s list the medicines that are grouped by timeline to instruct the user to take them on time and with the right dosage.
• • When the status of the drug turns gray with and has an ⚠️ icon, it means that the user has forgot to take the medicine. To confirm that you have taken that medicine on time, the user needs to click on it (or click Check all), then the status will turn green with a ✅ icon.

Medications Screen:

• • The main feature of this screen is to control the medicines list and medicines information.
• • User can manually enter the drug to be taken by entering the fields in the Manual section.
• • Or user can take a picture of a prescription, and the system will automatically add a list of medicines to take with a schedule based on the information found on the prescription.
• • If the list created from the photo is incorrect, the user can manually edit or delete it.

#7. Future features and functions

• • Support more languages: When taking a picture of a prescription, the application automatically processes the information. But currently, this feature only works when the prescription is in English. With other languages such as Vietnamese, Japanese, the application cannot handle it yet.
• • Save prescriptions for reuse: To make it easier to manage, we need to group medicines under a specific prescription. Also, in the future we may get same disease again, it would be much easier if we already have a prescription.
• • Online order medicine directly on the App: combined with the created prescription, users might no longer need to go to the pharmacy to buy medicine. Just order medicine through the app. The benefit of this is not only convenience, because if we can guarantee the origin and quality of the drug, as well as the price, then users will always feel secure when taking the medicine ordered through our application.
• • Extract the total amount/total number of days to take a certain drug: Most prescriptions have this information, such as 10 pills in total. Or the total number of days is 7 days.
• • Custom time setting feature: For each person, the time to take medicine is very diverse. So we can let users set their own time according to their habits.

#8. App limitations

Most of the limitations of the application lies in the ability to automatically generate data from the user’s photos. Specifically, it is lacking in points such as:

• Only English-language prescriptions can be processed.
• AWS Comprehend Medical could also missing some information on the prescription if their database doesn’t have the related data. For instance, it’s a new medicine brand/product so the data is not exist.
• It is not possible to understand all cases of schedule information. Because the time schedule for taking the medication is inferred by the application from the information in the prescription. There will be cases where it has not been programmed to handle.

#9. Summary

Easy Medicine is an idea that can be considered quite practical and useful. Its feature takes advantage of AI technology so it’s very easy to use. Hopefully in the future, this application can actually be deployed and operate.

#10. Links & references

• Detect text in images  |  Cloud Vision API  |  Google Cloud
• Detect entities (Version 2) – Amazon Comprehend Medical
• File trình chiếu – EasyMedicine