Project Analytics

The Analytics tab of your Project shows event-based analytics for your Project’s tasks, labels, and users. The Analytics tab has the following views:

• Tasks: View analytics on specific tasks in your Project.
• Labels: View analytics of labels in your Project.
• Collaborators: View collaborator performance in your Project.
• Issues: View analytics relating to issues in your Project.

The analytics available on the Analytics dashboard vary based on user roles:

• Admins and Team Managers have access to the Tasks, Labels, and Collaborators views, offering a comprehensive overview of team performance.
• Annotators, Reviewers, and Annotator + Reviewer roles can only view the Task Actions, Time Spent, and Label Actions tables, limited to their individual contributions.

Actions performed in the Label Editor or using the SDK are only tracked on the Analytics dashboard if they the tasks are actioned (submitted/rejected/approved).

Project Analytics have the following filters:

Filter	Description
Collaborators	Filter by specific Project collaborators.
Datasets	Filter tasks based on the dataset they belong to.
File name includes	Filter by file name. Regex is supported, allowing filtering by prefix or infix in the file title.
Event source	Filter by the source of the event. Either SDK or UI.
Workflow stage	Filter tasks based on their current stage in the Workflow.
Class	Filter by Ontology class.

Tasks

The Tasks tab of the Analytics dashboard provides a detailed overview of task throughput, stage efficiency, and task timers. It can help answer questions such as:

1. How productive was each collaborator in terms of labeling and reviewing tasks over the last week?

   • By filtering the data by Collaborators and Date time range, you can see how many tasks each team member worked on and how much time they spent on labeling and reviewing.

2. Which Dataset has the most labels added, edited, or deleted in a given Workflow stage?

   • You can filter by Datasets and Workflow stage to see which Dataset is being worked on the most and how many labels are being modified at each stage of the process.

The following charts are visible in the Tasks view:

Chart	Description
Task actions	Displays the number of annotation tasks submitted, skipped, review tasks approved, and review tasks rejected over a selected date range.
Time spent	Shows the total time spent actively working in the Label Editor, providing insights into productivity.
Task performance	Shows the actions taken for all tasks in the Project.

The Task performance table includes the following columns:

• File name: The name of the file associated with the task.
• Data type: The type of file used in the task (example: image, video, DICOM).
• Dataset: The Dataset from which the task originates.
• Total time: The total time spent on the task.
• Total created labels: The number of labels created in the task.
• Total edited labels: The number of labels that were modified in the task.
• Total deleted labels: The number of labels that were removed in the task.
• Time spent - The amount of time spent in a specific Workflow stage.
• Collaborators - The users who contributed to a specific Workflow stage.
• Created labels - The number of labels created in a specific Workflow stage.
• Edited labels - The number of labels modified in a specific Workflow stage. In review stages, this refers to Edit Review.
• Deleted labels - The number of labels removed in a specific Workflow stage.

Labels

Label analytics are not supported for Text and Audio modalities.

The Labels tab in the Analytics dashboard provides a detailed overview of your team’s labeling, reviewing, and task productivity, including time spent. It can help answer questions such as:

1. How many labels were submitted, approved, or rejected by the team over a given period?

   • Use the Label actions chart and apply the Date time range filter to view the total number of labels submitted, approved, or rejected within the selected time frame.

2. What actions have been taken for specific objects and classifications in the Ontology?

   • Refer to the Objects and classifications actions chart and expand classifications within the Ontology column to see detailed statistics for each classification answer.

3. How does the team’s productivity in labeling compare across different objects or classifications? Do certain objects take more time to label than others?

   • Analyze the Created, Approved, and Rejected columns in the Objects and classifications actions table to identify objects or classifications that might require additional review or clarification using their Rejection rate.
   • Compare the average time spent per object or classification by utilizing time-tracking metrics alongside these productivity statistics.

All charts allow you to view instance labels or frame labels by selecting the appropriate option from the dropdown next to the filter button.

Chart	Description
Label actions	Displays the number of labels submitted, approved, or rejected.
Objects and classifications actions	Shows the actions taken for all objects and classifications in the Ontology.

The Objects and classifications actions table includes the following columns:

• Ontology: Represents the Ontology class, encompassing both objects and classifications. For classifications, you can expand to view statistics for each classification answer.
• Created: Displays the total number of instances created for this Ontology class. Each instance is counted only once, ensuring that resubmissions of the same label are not double-counted.
• Approved: Displays the total number of instances of this Ontology class that have been approved. Approvals are counted multiple times if a label is approved in multiple review stages or if the task is reopened and reviewed again. Use stage-specific filters to see approvals per review stage.
• Rejected: Displays the number of instances of this Ontology class that have been rejected. Rejections are double-counted if a label is rejected in multiple review stages, rejected again within the same stage, or if the task is reopened and rejected again. Use stage-specific filters to see rejections per review stage.
• Rejection Rate: Calculates the rejection rate percentage of the given Ontology class by dividing the number of rejected labels by the total number of reviewed labels.

Collaborators

The Collaborators in the Analytics dashboard provides a detailed view of time spent annotating and reviewing by Project collaborators. It can help answer questions such as:

1. How much time did each collaborator spend on annotation / review tasks?
   • Use the Time spent chart to see the time distribution for each collaborator across annotation and review tasks. The Annotators and Reviewers tables, which provide total and average times for each collaborator.
2. Which collaborator spent the most time annotating or reviewing tasks in the Project?
   • Analyze the Time spent chart to identify the collaborator with the highest time allocation.

Chart	Description
Time spent	Displays the distribution of time spent per collaborator per day
Annotators	Table view of all relevant task/label actions and timers for each collaborator in the Annotation stages
Reviewers	Table view of all relevant task/label actions and timers for each collaborator in the Review stages

• Labels refer to objects and classifications.
• Approve actions are are double counted if there are multiple review stages.
• Review actions are double counted if multiple review stages are present or tasks get rejected again in the same review stage.

Both tables and all CSV exports are filter-sensitive; they only display information within the selected filter conditions.

The Annotators and Reviewers table columns vary depending on whether the Instance labels or Frame labels view is selected. They include the following columns.

Instance Labels

The Annotators table includes the following columns:

• Submitted tasks: Total tasks submitted by the annotator.
• Skipped tasks: Total tasks skipped by the annotator.
• Approved tasks: Tasks submitted by the annotator that were approved in subsequent review stages.
• Rejected tasks: Tasks submitted by the annotator that were rejected during review.
• Task rejection rate: Percentage of the annotator’s submitted tasks that were rejected. If multiple review stages are present, use workflow filters to view stage-specific rejections.
• Created labels: Total new labels submitted by the annotator. Include any pre-labels imported using the SDK by admins.
• Edited labels: Total existing labels edited by the annotator. This includes pre-labels from an Agent stage, or labels from a previous Annotate stage. Vertex / coordinate changes are not tracked.
• Deleted labels: Total existing labels deleted by the annotator. This includes pre-labels from an Agent stage, or labels from a previous Annotate stage.
• Approved labels: Labels submitted by the annotator that were approved during review.
• Rejected labels: Labels submitted by the annotator that were rejected during review.
• Label rejection rate: Percentage of the annotator’s submitted labels that were rejected during review.
• Total annotation time: Total active time spent annotating in the Label Editor, rounded to the nearest second.
• Avg time per task: Average time spent on each submitted annotation task. Calculated using the total active time spent in the Annotate stage divided by the number of submitted tasks.
• Issue actions against user: The number of Issue actions taken against the user.

The Reviewers table includes the following columns:

• Approved tasks: Number of tasks approved by the reviewer.
• Rejected tasks: Number of tasks rejected by the reviewer.
• Task rejection rate: Percentage of reviewed tasks that were rejected by the reviewer.
• Created labels: Total labels created by the reviewer using Edit Review.
• Edited labels: Total labels edited by the reviewer using Edit Review.
• Deleted labels: Total labels deleted by the reviewer using Edit Review.
• Approved labels: Number of labels approved by the reviewer.
• Rejected labels: Number of labels rejected by the reviewer.
• Total review time: Total active time spent reviewing in the Label Editor, rounded to the nearest second.
• Avg time per label: Average time spent on each reviewed label. Calculated using the total active time spent in the Review stage divided by the number of reviewed labels.
• Avg time per task: Average time spent on each actioned review task. Calculated using the total active time spent in the Review stage divided by the number of actioned reviews.
• Issue actions by user: The number of Issue actions taken by the user.

Frame Labels

The Annotators table includes the following columns:

• Submitted tasks: Total tasks submitted by the annotator.
• Skipped tasks: Total tasks skipped by the annotator.
• Approved tasks: Tasks submitted by the annotator that were approved in subsequent review stages.
• Rejected tasks: Tasks submitted by the annotator that were rejected during review.
• Task rejection rate: Percentage of the annotator’s submitted tasks that were rejected. If multiple review stages are present, use workflow filters to view stage-specific rejections.
• Created frame labels: Total new labels submitted by the annotator. Include any pre-labels imported using the SDK by admins.
• Edited frame labels: Total existing labels edited by the annotator. This includes pre-labels from an Agent stage, or labels from a previous Annotate stage. Vertex / coordinate changes are not tracked.
• Deleted frame labels: Total existing labels deleted by the annotator. This includes pre-labels from an Agent stage, or labels from a previous Annotate stage.
• Approved frame labels: Labels submitted by the annotator that were approved during review.
• Rejected frame labels: Labels submitted by the annotator that were rejected during review.
• Frame label rejection rate: Percentage of the annotator’s submitted labels that were rejected during review.
• Total annotation time: Total active time spent annotating in the Label Editor, rounded to the nearest second.
• Avg time per task: Average time spent on each submitted annotation task. Calculated using the total active time spent in the Annotate stage divided by the number of submitted tasks.

The Reviewers table includes the following columns:

• Approved tasks: Number of tasks approved by the reviewer.
• Rejected tasks: Number of tasks rejected by the reviewer.
• Task rejection rate: Percentage of reviewed tasks that were rejected by the reviewer.
• Created frame labels: Total labels created by the reviewer using Edit Review.
• Edited frame labels: Total labels edited by the reviewer using Edit Review.
• Deleted labels: Total labels deleted by the reviewer using Edit Review.
• Approved frame labels: Number of labels approved by the reviewer.
• Rejected frame labels: Number of labels rejected by the reviewer.
• Total review time: Total active time spent reviewing in the Label Editor, rounded to the nearest second.
• Avg time per frame label: Average time spent on each reviewed label. Calculated using the total active time spent in the Review stage divided by the number of reviewed frame labels.
• Avg time per task: Average time spent on each actioned review task. Calculated using the total active time spent in the Review stage divided by the number of actioned reviews.

You can choose which columns should be displayed and included in the CSV export using the Columns selector.

Issues

The Issues tab in the Analytics dashboard provides a detailed view of all issues created in the Project.

Chart	Description
Issue actions	Shows the number of issues created, resolved, and opened.
Issue tag occurrence	Shows the count of all issue tags used within the chosen timeframe.
Issue tag occurrence by class	Shows the actions taken for all objects and classifications in the Ontology.

The Issue actions chart shows the following:

• Total issues created: The total number of issues created.
• Total issues resolved: The number of issues marked as resolved.
• Total issues reopened: The number of issues that were reopened.

Example - How to Understand Issue Analytics:

If your Issue actions chart shows:

• Total issues created = 19
• Total issues resolved = 20
• Total issues reopened = 2

This means that:

• 19 issues were created.
• All 19 issues were resolved.
• 2 resolved issues were reopened. (This includes manually reopening an issue and re-rejections of labels)
• 1 of the reopened issues was resolved.
• There is one remaining open issue.

---

Analytics FAQ

How do I track annotator performance on pre-labeling/model labels?
Annotator performance on pre-labeled or model-generated labels can be tracked using the Edited Labels and Deleted Labels metrics in the Annotators table. These include:

• Labels that were pre-labeled using an Agent stage.
• Labels that were modified by annotators during the annotation process.

Vertex/coordinate changes are not tracked as edits.

---

Can I export analytics for payroll or performance tracking?
Yes, CSV exports from the Analytics tab provide data that can be used for payroll or performance tracking.

• The Annotators and Reviewers tables contain total time spent, tasks submitted, approved, and rejected.
• Data in exports is filter-sensitive, so you can apply filters before exporting.
• Review time is rounded to the nearest second per task.

---

What is counted as a created label?
A created label is any new label added by an annotator or reviewer. This includes:

• A completely new label drawn or added during annotation.
• Labels created in Edit Review mode.
• Labels submitted using the SDK (counted under the annotator who submitted the task).

---

What is counted as an edited label?
An edited label is a modification made to an existing label. This includes:

• Changes to attributes or classifications.
• Labels modified from pre-labeling or a previous annotation stage.

Changes to vertices/coordinates alone do not count as an edit.

---

Can I customize what data is included in the CSV exports?
No, CSV exports are pre-defined and contain all available metrics. However, you can filter the data before exporting to limit what is included.

---

How are timers tracked?
Timers track active time spent in the Label Editor:

• Time is rounded to the nearest second per task.
• Pauses or inactivity do not count toward the active time.
• The Avg time per task metric is calculated by dividing the total annotation/review time by the number of submitted tasks.

---

Are label events counted if a task is skipped?
No. If a task is skipped, any actions performed before skipping (for example, creating/editing labels) are not recorded in the Analytics dashboard.

---

How are actions outside of the task queue counted?
Actions performed outside of the task queue, such as label modifications using the SDK are only tracked if the task is subsequently submitted, rejected, or approved.

---

How does the SDK filter work?
The Event Source filter allows you to view actions performed using the SDK or UI.

---

What happens if someone is inactive or takes a break?

• Inactive time is not recorded. The system only tracks time when the annotator is actively working in the Label Editor.
• If an annotator takes a break without closing the task, their session remains open, but inactive time is not counted.

---

How long does it take for actions to appear on the Analytics dashboard?

• It takes a maximum of 5 minutes for any action to appear on the Analytics dashboard.