Realtime Onscreen Translator App with AI [greenfield]


The client is an industry-leading Asian development company. Their goal was well-defined: to create a real-time screen translation app that utilized AI technology to read all text on a user’s screen at all times, translating it from the original text to a user’s preferred language as instantly as possible. The client had a clear and concise idea about what they wanted to create and was simply in need of a development team who would turn their vision into a reality.



Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project
Screenshot of a previous project


Objective


To build a cross-platform (MacOS, Windows) desktop app for onscreen translation. The goal is to keep all context of the original text as accurate as possible, which is achieved by drawing back all text to its original place on the screen, just in the user’s preferred language.



Process


1,The client planned the application from a business perspective.

2,Our teams worked together to plan the app from a technical perspective.

3,Our team developed several prototypes and iterated through designs and possible technologies until we reached a capable solution that ticked all the boxes.

4,We created the application and handed it over to the client for release.

5,We now provide ongoing support for new features and fixes.

6,We have a regular weekly meeting with the client where we discuss newly implemented features, upcoming feature requests, and bug fixes.



Scope


Desktop App Development

Microservice-based Backend Development



Tech stack


Typescript

MacOS

Windows

Python

NodeJS

OCR

Tensorflow

C++

Objective-C

Docker



Technical challenges


To find the adequate tech stack and AI models for our needs

To find ways to run AI models in the browser, if required

To create a performant, universal solution that is platform agnostic

To code missing modules and functionality in Native (Objective C++ for Mac and C++ for Windows)

To provide an application speed of around 1FPS or more for frequent updates when on-screen text changes or updates, such as side-scrolling text bars, live stream chats, etc.)



Outcome

We worked hard to ensure our team delivered the application on time while fulfilling all client expectations. We ensured we did, and the client was handed a fully functioning Desktop app (with the corresponding backend) by the agreed deadline. As we've developed a cross-platform app, we could really quickly iterate over new designs, features, change requests, and so on, saving both money and time for the client and allowing ourselves to carry out the work in the most efficient way possible. Moreover, most of the codebase is similar between platforms, so it's easier and cheaper to maintain. The only part of the codebase that varies between the Mac and Windows versions is the platform-specific logic that was inevitable to code in native.

By clicking 'Accept' you agree to the use of all cookies as described in our Privacy Policy.

© 2023 Scriptide Ltd.

All rights reserved