NOVA - Service Outage

I’m reaching out to let you know that after much work by the Lightcast staff, we have fully recovered from the cyberattack we suffered on March 15. All products are back on line and all job postings data are up to date.

This has been a challenging experience for our clients and all of us at Lightcast. I want to thank you for your patience and support during this period. I also want to tell you again how deeply sorry all of us at Lightcast are for the disruption to your services. We know how much you rely on our data and our products, and we want to keep that trust.

As you may have already seen, we have produced a report on the incident and the investigation by our outside cybersecurity consultants. The full report is worth your attention, but I wanted to share some important conclusions with you:

• No further malicious activity has been detected since March 15;
• An outside forensic investigation has found no evidence of a data breach, staging, or exfiltration;
• All affected services have been restored on new data infrastructure; and,
• We are confident it is safe to connect to Lightcast services.

The most important conclusion, however, is that we’re going to work even harder to provide you with the best possible service and insight. We learned a lot and those lessons will be put to work in how we maintain security, provide service, and communicate with clients.

Please don’t hesitate to reach out to us with any questions or concerns. And once again, thank you for being a Lightcast client.

Sincerely,

Chris Kibarian, CEO

A final report on the March 14 security incident is available on our outage-update page. Please contact your customer service representative for additional details.

Global Postings API and Snowflake Postings Global have been restored. Restoration of BG Cloud and certain custom data pulls are still ongoing. Please contact your customer service representative for additional details.

Canadian Job Posting feeds have been processed and are up to date in Lightcast products.

Canadian feeds, except for the monthly CSV feed, have been restored. The CSV feed is expected to be restored in the next several days. Please contact your customer service representative for additional details.

Several additional data feeds have been or are expected to be restored this week for global customers:

• ANZ 3.4.5 is restored
• ANZ 3.4.3 is expected to be restored April 19
• SGP 4.11 is expected to be restored April 19

Standard feeds power dashboard products, like Analyst, and most APIs. Custom feeds are designed to meet the needs of a particular customer. Custom feeds in Australia have been restored.

Canadian standard feeds are expected to be restored by April 24, with custom feeds restored by May 2.

All UK standard feeds have been restored, and Analyst, API, and Snowflake are up to date. The recovery process continues for other global and custom products. Please contact your customer service representative for details about your product.

US standard data feeds are now restored and up to date. Our recovery process continues for global data and custom products. Please contact your customer service representative for details about your product.

US job posting processing continues, with standard NOVA and Analyst feeds expected to be caught up to March 20 as of today. Processing will continue over the weekend, but is proceeding more slowly than anticipated. We anticipate data will be up-to-date in standard Lightcast applications, such as Analyst, on April 3.

Full recovery dates for custom US applications will differ. Contact your customer service representative for details.

US job posting processing has started with updated files expected in some applications today. Full restoration is not expected until Friday.

FOCUS API and Career Insight service has been restored effective today (March 29).

We now have estimated timelines to restore job posting processing and LENS API parser data outside the US:

• In the UK and Canada, full restoration is expected on or before April 17
• In Singapore, Australia, and New Zealand full restoration is expected on or before May 1

These timelines are for standard feeds and work may progress faster than anticipated. For custom feeds, please contact your customer service representative.

US job posting processing and NOVA service is still anticipated to be restored on March 31.

In addition, we expect to restore service for FOCUS API and Career Insight users this week.

We now anticipate that US job posting processing will begin March 29. However, we still expect that the NOVA API and job posting processing in all US products will be caught up by the original date of March 31.

LENS APIs are functioning normally. We have made improvements to allow customers with backed-up workloads to process more quickly and catch up.

We restored service for the LENS API yesterday (March 21) and the service is operating normally. The next step will be restoring LENS Metered On-Premise service. We will provide a timeline for that later this week. In the meantime, LENS Unmetered On-Premise installations were not affected by the outage and we have been switching over Metered accounts to Unmetered.

As per our announcement yesterday, Lightcast is currently unable to process new job postings. None of our products have job posting data after March 13.

Our current estimate is that we will begin to provide updated U.S. job postings March 27, with complete restoration of U.S. NOVA service and job posting processing by March 31. Postings will be processed in order, with those closest to the incident processed first (March 13-15).

Service to the UK, Canada, Singapore, Australia and New Zealand will take additional time. We will have a more precise timeline later this week.

For more details on this outage - https://lightcast.io/outage-update

LENS API service has been restored. To access the restored service, new credentials will be required. Our team will be creating credentials and distributing them to clients starting at 3 pm ET/noon PT. Please contact your customer service representative for more details.

The rebuilt LENS service has entered production testing. We still anticipate providing credentials to LENS clients tomorrow (Tuesday, March 21). Our next update will be at 11 am ET/8 am PT Tuesday.

We are on track to restore LENS service Tuesday (March 21) with additional security and reliability measures in place. We will have additional updates later today.

Our estimated time for LENS recovery is being revised to Tuesday. The additional time is to put enhanced security and reliability measures in place. Our next update will be at 11 am ET/8 am PT on Monday.

Our testing process for LENS restoration is continuing. We are pushing back our estimated time for LENS recovery, and expect to issue new credentials to clients Monday afternoon ET. We will issue a further update Sunday evening ET.

We remain on track for LENS API restoration tomorrow night or Monday morning ET. We will provide a further update at 11 am ET/8 am PT tomorrow.

Our restoration plan continues to proceed on schedule. Today we are scaling and load-testing our rebuild of the LENS APIs and continue to expect to issue credentials Sunday night or Monday morning, Eastern Time. Our previously released workarounds for NOVA and LENS Metered On-Premise remain available.

All other products are operating normally, however, analysis that includes data since March 15 is likely to be unreliable because of lags in posting data.

We remain on track for a restoration of LENS API service on Sunday night or Monday morning Eastern Time. Your customer service representative will reach out to you with credentials and any other assistance you need, including workarounds for clients using NOVA and LENS Metered On-Premise.

All other products continue to operate normally but are experiencing lags in their posting data.

Thank you for your patience as we work to resolve these problems. We anticipate providing a further update at roughly 11 am ET/ 8 am PT Saturday.

We continue to work to restore service to LENS and NOVA customers.Our support teams continue to be ready to assist clients in implementing workarounds for NOVA and LENS Metered On-Premise. LENS APIs remain off-line.

Clients using our NOVA workaround should know that this process only provides pre-outage data and not current or ongoing data.

Job posting collection has been proceeding normally; the interruptions only affect processing. Any data lags in Lightcast products will be restored once all problems have been resolved.

We will provide updates daily at 11 am ET/8 am PT and at 10 pm ET/7 pm PT.

We’re experiencing an extended outage at one of our data centers. We are working aggressively to restore service. Impacted services include NOVA feed delivery, LENS Metered On-Premise installations degraded, LENS API, and Spectrum API. Workarounds are available for NOVA and LENS Metered On-Premise, so please contact customerservice@lightcast.io for details.

All other products are functioning normally, but may experience delays in receiving posting updates.

Any previously scheduled data fixes to products will be delayed until after this issue is resolved.

We know you rely on Lightcast data, and we take that responsibility seriously. We deeply regret the impact on our clients. We are taking every possible step to restore service as quickly as possible.

Products currently impacted include LENS API (Spectrum, X-Ray and Optic), LENS Metered On-Premise, and NOVA Feed Delivery. Our alternate NOVA Feed Delivery server is available - please reach out to customersupport@lightcast.io for credentials).

There has been an unexpected service outage impacting Burning Glass products, including LENS (API and Metered On-Prem) and the NOVA Feed Delivery.