Recomendados
Más contenido relacionado
Similar a Building Intelligent Workplace Limits and Challenges RIGA COMM 2023
Similar a Building Intelligent Workplace Limits and Challenges RIGA COMM 2023 (20)
Último
Último (20)
Building Intelligent Workplace Limits and Challenges RIGA COMM 2023
- 1. Building Intelligent Workplace: Limits and Challenges Lessons learned from ML / AI enabled solution for better workplace RIGA COMM 2023
- 2. Speaker 2 Muntis Rudzitis LEAD DATA SCIENTIST EMERGN Bio: • Machine Learning Lab • Technical architect for AI+ML, data enabled solutions • Experimentation, prototyping • Python, Azure, Power BI • Speaker
- 3. • Intro • Intelligent Workplace • ML tasks & Challenges • Lessons learned 3
- 4. Intelligent Workplace • R&D work done in field of text analytics and processing - Notification analysis - Next-best activity - Meeting analysis - Similar tasks • https://www.emergn.com/thought-papers/insights-on-intelligent- automation-of-knowledge-work/ • https://www.emergn.com/insights/smarter-approach-notifications-ml-ai/ • https://www.emergn.com/insights/why-process-mining-is-the-top-skill- to-learn-for-business-analysts-in-2022/ 4
- 5. The knowledge work spectrum 5 TASK EXECUTION WORKFLOW MANAGEMENT DECISION- MAKING Copy Enter Sort Translate Classify Extract Prioritize Analyze Decide With typical activities
- 6. Work area 6 • Of the different classes of demand related to developing and then running a product or service, the Intelligent Workplace is aimed at Operations Demand. • This kind of work is usually known ahead of time – or is knowable. • It could, however, be planned or unplanned. • Examples of places where this kind of work happens: • Help Desk • Call Center • Warehouse • A shop QUITE A BIT OF CERTAINTY Work structure
- 7. Problem we are solving: Goal Alignment What should the focus be? Is the work easy or hard? How are we using capacity? Objectives and goals (Top-Down) Smart working environment Contextual work items (Bottom-up) Managers Workers
- 8. Intelligent Workplace concept I I I I Insights available (reusable knowledge) 15 min 245 2 days ago Average time per task Similar tasks done this week Last time you did similar task Description available Colegues to ask about it: • George • Fred • Austris Workarounds: • 1 • 2 • 3 Write feedback Best/worst case: • ACME • Ministry of Finance • If it is unique – most problably no knowledge there • If it is repeating – there are at least statistics + insights could be collected • On-prem + When it is usualy done: 9:00 – 12:00
- 9. Intelligent Workplace – visualize work Insights available (reusable knowledge) 15 min 245 2 days ago Average time per task Similar tasks done this week Last time you did similar task Description available Colegues to ask about it: • George • Fred • Austris Workarounds: • 1 • 2 • 3 Write feedback Best/worst case: • ACME • Ministry of Finance Work done – New type of analytics 5000 245 Number of taks done today Task 1 Task 2 Task 3 Tasks with feedback Score A
- 10. • Task list view • Prioritized • Aggregated • Intelligent context sensitive help Source system task metadata Task-level recommendations (similar tasks, known standards or solutions) Topic level help with SME/colleague contacts Analytics, statistics and estimates Intelligent Workplace interface: Worker’s View
- 11. • Statistics • Task information • Similar tasks • Who can help • Past work examples • Reviews • Workarounds and additional materials Analytical insights for the Worker
- 12. Intelligent Workplace interface: Manager’s View • Visualizing work • Structure knowledge • Improve process • Day-to-day progress and problems • Plan ahead based on analytics From "people" manager to work manager/coordinator. Role of an operator, observer, coordinator, productivity enabler.
- 13. • Intro • Intelligent Workplace • ML tasks & Challenges • Lessons learned 13
- 14. • Statistics • Task information • Similar tasks • Who can help • Past work examples • Reviews • Workarounds and additional materials • Overall goals to improve productivity through knowledge reuse/accumulation: • Reduce the number of unique tasks • Apply knowledge to optimize repetitive tasks • Reuse knowledge for repetitive tasks Tasks to solve
- 15. Task similarity • Task similarity is a tricky • In defined processes: • point of reference • known steps and context • potential solution overlap • In undefined process: • no clear point of reference in process • very varied and significantly differ even when looking similar at high level • context often overshadows the tasks underneath; goals can end up needing similar steps to complete 15 TASK EXECUTION WORKFLOW MANAGEMENT DECISION- MAKING
- 16. Task similarity: Concept • Knowledge/support work or its artefacts are often semi-structured/unstructured. • Key aspect here – find distance/similarity metric for tasks (or people) • Some of the metrics we experimented with: - Average time per task - Estimated task time - Last time you did a similar task - Colleagues to ask about the task - Similar tasks - Best / worst cases 16 IWS Description Time Person Documents Forms Insights, recommendations, help, productivity ...
- 17. Task similarity: Data Source Dataset: • Jira dataset • Real data • 66 Open-source Apache projects (used only select few) • Each project has about 2000 tickets • Feature set is closer to reality • Calculated metrics are more believable • Experimented with local Ops dataset https://zenodo.org/record/3942332
- 18. Task similarity: Solution 18 IDSummary Near_summary Near_ID distance 12507134Create a LOGO for Airavata Project Airavata Logo Drafts 12901695 0.7933585 12507134Create a LOGO for Airavata Project Improve Logo and Banner text for Airavata 12653988 0.7009898 12507134Create a LOGO for Airavata Project Airavata Designer Guide 12675281 0.639134 12507134Create a LOGO for Airavata Project Update Airavata Logo Website Banners and Branding 12895578 0.626145 ...... ... .. 0.6241394
- 19. 19
- 21. Task similarity: results & Lessons learned 21 • Limit the context & scope • Repeatable – so that results overlap & help • Data evolves over time • Useful but can't 100% rely on Writing style variance Lessons learned Different people Different teams Different projects … … Different technologies Writing style variance
- 22. Using: • Neural Networks • Support Vector Machines • Random Forests 22 Major Critical Minor 63% Achieved accuracy Issue priority / story points
- 23. 23 Error distribution Confusion matrix for test https://arxiv.org/pdf/1609.00489.pdf Issue priority / story points
- 24. Lessons learned • Uneven data scale • Initially easy/fast on high level for humans, when doing top-down • Difficulty often comes form technical nuances, requires data from different contexts to compare • Data is fairly private 24 Lessons learned Confusion matrix for test
- 25. Learning from mistakes aka “what would I do differently Today?” • Data availability: lots of Dev not Ops - Problem framing • Flip the problem: - Codex, Llama 2 (CoPilot, ChatGPT, Code Llama, LangChain) etc. - Enrich data while its fresh - Enrich data when its finished • Better local models availability • Limit scope differently • Worth revisit, proven success in similar tasks 25
- 26. 26
Notas del editor
- Done research in area If you follow us on social media
- To paint the landscape.. How we see work - Knowledge work spectrum Part of the work is processing, (in Black) - Reading documents, writing Emails, Book meetings / zoom Work that does not need much of the education or skillsets. And then there is the knowledge work (in Red) - requires, discussion, analysis, collaboration and decision. Like making this PPT. More time per task. (RE)USE vs CREATE Teacher reference We consider both these types of work is the way to create better experiences and better products. Leaves DATA TRAIL More on left, more data. Better ML.
- Regulatory areas have processes, easier to base yourself Insurance, government etc
- For manager to help From manager to operator, observer, coordinator
- Covey, Eisenhower popularized Urgent/Not Urgent and Important/Not important work seperation/prioritization
- Not talking about big data UNDERSTANDING WORK UNDERSTANDING DATA DATA LEAVES FOOTPRINT
- Task similarity is tricky metric to calculate OPS Development For the undefined process, the tasks to do can be very varied and significantly differ even when doing similar looking task at high level.
- https://ai.googleblog.com/2018/05/advances-in-semantic-textual-similarity.html https://tfhub.dev/google/universal-sentence-encoder/4 And what do you get as an output?
- Cant really show the data.. Limit the context & scope (picture) Repeatable – so that results overlap & help, otherwise very little help, so Ops better Data evolves over time – tickets get more data, status, time completed etc Useful but can't 100% rely on successfully used in other use cases like survey analysis
- ServiceDesk, Jira etc – smart plugins and features appearing MS Outlook showing relevant docs for meetings