Project Evaluation Plan

Effective evaluation is important for assessing the achievement of project goals, evaluating team and solution performance, optimizing resource allocation as well fostering continuous improvement throughout the project.

‍

Project teams develop and use an evaluation plan in order to:

Ensure the project team has agreed to the KPIs used and methods selected to inform the different partners in the project to meet funding reporting needs.
Set up any tools, services, and access controls for monitoring credential issuing, use, and verification across system environments and use contexts.
Monitor the necessary activities and engagements throughout the life of the project to ensure proper operation, fix problems, improve outcomes, and fulfill reporting needs.

Evaluating Systems for Performance

Monitoring well-selected key performance indicators (KPIs) enables you to evaluate the effectiveness of a project issuing LERs as VCs for SBHA use cases and activities. A wide variety of KPIs could be used, but the most effective will be metrics that help you identify, diagnose, and solve problems over time. We’ve provided a list and some guidance to assist your project team in identifying which KPIs are the most valuable for evaluating your project performance.

‍

A couple tips:

Cohort-based tracking methods enable you to learn better than sub totals of award or user counts. For example, discovering that a majority of job-seekers quickly become inactive lets you identify and solve usability problems they encountered, whereas a simple incrementing counter of total signups would miss a major problem.
Avoid collecting personally identifiable data as part of KPIs as much as possible, especially when it is not needed to make KPIs actionable. When user identifiers are necessary for correlation or cohort-based metrics, consider using generated identifiers resistant to correlation with original users wherever possible

‍

Specify how you will monitor and evaluate your project's performance, complete the templatewith the guidance of the table below.

Template

KPI	What it monitors and tracks	What it indicates
VC Issuance Rate	Measure the rate at which LERs are issued as VCs over the project's lifetime. Calculate the number of VCs awarded per day and the number of recipients per day. For some projects, tracking by week may be more informative.	This KPI reflects the adoption and utilization of the credential-issuing system over the project's lifetime. It may help represent the effectiveness and growth of the issuing processes or the removal of friction.
Time to VC Issuance	Measure the average time it takes to issue LERs as VCs, from the completion of requirements to when the recipient has obtained the VC, such as into their digital wallet.	This KPI evaluates the efficiency of the credentialing process and can help to identify friction points in the business activities or processes.
Email Open Rate	Open rate by stakeholder type and email type including credential issuance, training, project- or partnership-related emails, reminder emails, informational/"marketing" emails, etc.	May help indicate the level of interest in LERs, the effectiveness of subject lines, or stakeholder capacity.
VC Revocation Rate	Track the number or percent of VCs revoked over time. Organizing revocation events by revocation date or by issuance date of the revoked credential may make this data more actionable.	The KPI indicates an issuer agency revoking a credential as well as the possible reasons for revocation.
VC Accuracy	Assess the accuracy of information contained in verifiable credentials, such as educational achievements, skills, work experience, and certifications.	This KPI ensures data integrity and reliability and, through validation checks, can improve the propensity of using a LER issued as a VC.

Evaluating LER VCs for Impact

Implementation projects have the potential to impact the local, regional, and state ecosystems within which they are implemented. We’ve provided here a summary list of possible indicators organized by the ecosystem partner group that could be used to evaluate the wider impact of your project on the ecosystem. When evaluating impact, ensure you are considering metrics that truly help you measure progress toward achieving your project goals. (Don’t forget to review your Project Strategy template, if you choose to leverage that resource!)

Specify how you will evaluate your project's adoption and impact. Reflect on the table and complete the template.

Template

Measuring Adoption
KPI	What it monitors and tracks	What it indicates
Earner Engagement	Monitor counts of workers/learners using the solution monthly to track learner engagement with each relevant VC technology solution or platform over time.	This KPI assesses Learner and Worker adoption, engagement, and success. Cohort based analysis, to compare activity levels across groups of people who signed up at the same time, enables projects to identify inflection points or changes in retention.
Earner Acceptance Rate	Track the percentage of learners or workers who accept LERs that have been issued to them as VCs.	May indicate good accessibility, engagement, problem-solution fit, user training, and issuing effectiveness. Improvements may reflect increasing usefulness of credentials or removal of process friction.
VC Share Rate	Track the percentage of earned credentials that are shared by earners to at least one other system, platform, or person.	This KPI indicates credential portability and positive earner engagement with credentials.
VC Verification Rate	Monitor the rate at which LERs issued as VCs are successfully verified by employers, educational institutions, or other verifiers.	This KPI reveals the extent to which VCs are successfully circulating end-to-end within the network, even without knowing which VCs are being verified.

Measuring Impact
KPI	What it monitors and tracks	What it indicates
Skill Development and Acquisition Rate	Using a method relevant to the project, track the growth of skill recognition within the project context over time.	This KPI assesses the effectiveness of credentialing in promoting the development of particular skill sets.
Renewal Rate	Track the percentage of credentials that are renewed or updated over time to reflect ongoing learning, skills development, or changes in employment status.	This KPI indicates user engagement and currency of credentials.
Employment Placement Rate	If possible, measure the rate at which individuals who earned credentials are successfully placed in roles targeted by the project. (This may be difficult to measure, but it may be worth considering what this means to your project.)	This KPI reflects the employability and market value of credential holders.
Qualitative Feedback and Quantitative Satisfaction Scores	Gather feedback from partners, including learners, employers, credential issuers, and verifiers, to assess satisfaction levels, user experience, and perceived benefits of using verifiable credentials. Net Promoter Scores are a possible way of collecting and recording this metric.	This KPI provides qualitative insights into system effectiveness and user perceptions. Tracking this data over time enables the association of changes in user satisfaction with changing facets of the recognition environment.