Service Status

Our hosting provider has identified an issue with our database. It is currently running from the standby server. They will migrate to a new primary server this evening. Kanban may be unavailable for a period of time.

Edit 2020-11-20:

The migration is complete. There were a couple of minor issues that have been solved already. Thanks for your patience while we moved to this new improved infrastructure.

We're moving!

We have built new infrastructure at a well-known cloud hosting provider and are migrating customers to the infrastructure throughout this week. If all goes well, we'll be fully moved by the end of the week.

The change offers improvements in a few areas:

• The application has improved reliability at all layers- load balancer, app layer, cluster node layer, and data storage layer. Each layer has automatic failover built-in, limiting the impact of any failure.
• The new provider has much more fault-tolerant hardware and network and better monitoring, so certain types of problems that previously required action from us or our old provider will now be handled automatically.
• Resources can be provisioned much faster now (minutes instead of days) so we can handle capacity demands much more quickly.
• The new provider has data centers all over the world, which means in the future we may be able to offer you service in your region. (The new infrastructure is in New York City.)

How does this affect me?

Your Kanban application will be unavailable for about 10 minutes while we migrate it. We intend to do this during evening or night time hours for all customers.

If you're a ConnectWise on-premise customer who protects your ConnectWise server behind an IP-whitelisting firewall, please review the knowledgebase page about allowing access to your server.

Update 02:50 AM EST Nov 13

All Kanban web services are online now. PSA data sync jobs are catching up. If you see out-of-date data on your boards, it should catch up within about 30 minutes.

Update 01:25 AM EST Nov 13

The host has replaced the router and services are coming online again. Some of our monitoring continues to report offline customers but this is likely due to delay in monitoring checks. Please check your Kanban instance.

Update 10:50 PM EST Nov 12

The host has found that a top of cabinet router has failed. They are preparing to enable a standby router. We'll post when they give us an ETA.

Update 10:20 PM EST Nov 12

Our host is investigating the issue. We are preparing new infrastructure at another host.

Update 8:15 PM EST Nov 12

We continue to monitor the situation, and apologize for the impact. At this time we don't have an ETA for restoring service.

Update 6:19 PM EST Nov 12

There is a networking issue with our hosting provider. We have opened a support ticket with them and will be working with them to resolve this issue.

TopLeft is unavailable for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

TopLeft performance is unavailable for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

Update October 28th 2020:

ConnectWise support has said that the fix for this issue has been rolled out to all instances. If you are still experiencing this issue please open a support ticket by emailing help@topleft.team with two things:

1. the ID of a ticket you saw was delayed (choose any ticket)
2. a screenshot of the ConnectWise audit trail of that ticket around the time you saw the delay

This will allow us to confirm the issue is with delayed notifications and give you details you can send to ConnectWise.

----------------------------------------------------------------------------------------------------

Update October 20th 2020:

We have been informed by ConnectWise support that a temporary fix has been applied to fix this issue for North American cloud instances, and they are currently working on a permanent fix that should cover all cloud instance regions. From ConnectWise Support:

"We identified the issue in NA and made temporary steps to resolve that there. But our development team has updates to resolve this issue at the root and that is being applied to all of our environments so if any of the others were seeing this issue then this should resolve that as well."

----------------------------------------------------------------------------------------------------

We have reports from multiple ConnectWise Cloud (hosted by ConnectWise) customers that ticket updates from ConnectWise are not being shown in a timely manner. For example, when a ticket is created or a ticket status is changed in ConnectWise, the change takes up to 5 minutes to appear in Kanban.

In each case, the problem has been found to be delayed notifications of ticket changes from ConnectWise. We've seen notifications arrive anywhere from 10 minutes to 4 hours late.

Even with delayed notifications, updates shouldn't take more than 5 minutes to appear, because TopLeft performs a check for updated tickets every 5 minutes.

This applies only to ConnectWise Cloud customers. It doesn't affect on-premise ConnectWise customers or Autotask customers.

What should I do?

First, let us confirm the issue you're seeing is the result of delayed notifications. Email help@topleft.team with two things:

1. the ID of a ticket you saw was delayed (choose any ticket)
2. a screenshot of the ConnectWise audit trail of that ticket around the time you saw the delay

This will allow us to confirm the issue is with delayed notifications and give you details you can send to ConnectWise.

If the cause of the problem is delayed notifications, please submit a support ticket to ConnectWise. You can use the following wording- replace the marked values with your own words:

We use a third-party tool for visualizing ticket data that integrates with ConnectWise Manage using the ticket callback API. Since [replace with date when you began seeing delays] the data has been slow to update because ticket callbacks are being delivered slowly. Until recently everything had been updated quickly.

Attached is a screenshot of received callbacks showing delayed delivery. [attach the screenshot and details we provide]

This is impacting our team's ability to work effectively. Please investigate why ticket callbacks are being delivered slowly.

Is there a workaround?

TopLeft has 2 options that may reduce the impact of this problem for you. Administrators can enable these options.

From a Kanban board, in the main menu click Admin > Options.

On the options page, enable one or both of these options:

DJCONNECTWISE_ONE_MINUTE_SYNC - this performs a query for updated tickets every minute instead of every 5 minutes, reducing the time to show ticket updates in TopLeft when notifications are delayed.

SHOW_CARD_SYNC_FUNCTION - this enables a "Sync Ticket" option in the "..." menu of each ticket, allowing users to trigger TopLeft to fetch any updates for that ticket from the ConnectWise server.

To enable those options, click the checkbox in the Value column and click Save at the bottom of the page.

Be aware the KANBAN_REFRESH_INTERVAL option doesn't change how often data refreshes from the ConnectWise server; it changes how often the board web page refreshes with data already on the TopLeft server.

What is TopLeft doing?

On October 5 TopLeft reached out to ConnectWise platform support to make them aware of the problem.

Update 2020-10-23 14:52: We have rolled-out the fix for this issue. TopLeft embedded mode should be available when using chrome again. Please open a support ticket if you still experience any problems with this.

=======================================================================================================================================================================

Recent updates to Google Chrome have made CW Embedded mode unavailable. We have prioritized a fix for this and will update this post with an ETA for that when available.

In the meantime we suggest you use the TopLeft application from your url, i.e. https://.topleft.team . If you require embedded mode for your workflow it is still available in Firefox.

Update 2020-10-20 17:04: There was an issue in our hosting provider's infrastructure. Access to TopLeft has been restored to all clients.

TopLeft performance is degraded for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

TopLeft is performing maintenance on the main database server in order to improve performance and reliability. This is expected to take 15-30 minutes.

Update 2020-08-10 12:29: All instances have been restored.

====================================================================

Update 2020-08-10 11:15: There was an issue in our cluster infrastructure related to resource scheduling. The root cause has been identified and resolved and instances are coming back online as resources become available.

====================================================================

TopLeft performance is degraded for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

On July 9 between 6 PM and 10 PM Pacific Standard Time we will be performing maintenance on our service cluster to improve the reliability of the management plane. We don't anticipate downtime... but you know how these things go sometimes.

Update 9:24 AM PST - 2020-06-11

A scheduling problem caused a node to be overscheduled. The issue is resolved but we are looking into the root cause of the issue.

===========================================================================================================

TopLeft performance is unavailable for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

[Update 2020-05-19 15:23 PM PST]

We have applied the changes and full service has been restored to all instances.

-===========

[Update 2020-05-19 14:40 PM PST]

Some sites are unavailable again as we apply changes to improve distribution of the load in our cluster. This is temporary.

============

[Update 2020-05-19 13:35 PM PST]

The issue is resolved and we continue to investigate the root cause.

============

Kanban boards are unavailable for some customers. We are investigating an issue in our load balancer.

TopLeft will be performing maintenance on the Kanban application infrastructure on April 23, 2020, between 5 PM PST and 7 PM PST. Boards may be unavailable during this window.

We are updating networking components to make them more reliable and reconfiguring storage on certain cluster nodes.

Update: 12:34 PM PST

All instances have been restarted which has resolved the issue.

Some TopLeft instances are getting intermittent errors when dragging cards and receiving callbacks. We are currently investigating and will update the status when we know more.

TopLeft performance is unavailable for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

UPDATE 9:25 PM PST

Kanban service has been restore for some time. There seem to be some latent bad routes affecting some of our monitoring, making it more difficult to verify that all sites were working again.

UPDATE 6:07 PM PST

The problem is a network issue at our hosting provider, we are still currently investigating.

TopLeft is unavailable for all customers. We are investigating and attempting to bring the affected servers back online.

UPDATE 11:32 AM PST

Our hosting provider has resolved the issues caused by the power outage. Access to TopLeft has been restored to all clients.

UPDATE 11:10 AM PST

Our hosting provider in Seattle is experiencing a power outage and they are working to resolve it. This is causing routing problems for our TopLeft clients. We don't currently have an ETA for the resolution, but we will update this post no longer than one hour from now.

TopLeft performance is degraded for some customers due to a problem in our infrastructure. We're investigating and will post updates here.

Some TopLeft instances are getting intermittent timeouts. We are currently investigating and will update the status when we know more.

================================================================

Update: 2020/02/26 12:40

We tracked this down to reduced disk I/O due to a failed raid, and we have failed over to our warm standby.

[Update 2019-12-03 16:35 PST]
We identified the networking issue in our infrastructure and applied a fix. All Kanban boards are available again.

==============================

Kanban boards are unavailable due to a DNS problem in our infrastructure. We're investigating and will post updates here.

Update 12:33 AM PST: The fix has been applied. The embedded kanban service has been restored.

Update 11:22 AM PST: We are currently deploying the fix to all instances and expect this issue to be resolved shortly.

Accessing embedded CW Kanban tabs in ConnectWise is unvailable for all customers. We believe this to be a change to the ConnectWise API and are currently working on a solution. Current ETA of fix is under a few hours.

We'll be performing server maintenance on 2019-07-19 between 5 PM and 8 PM PST. CW Kanban might be unavailable for periods during this window.

[Update 2019-04-18 09:30 PST]
All sites are available. The cause appears to have been a network partition. All workloads have been moved off the unreachable host.
==============================

CW Kanban is unavailable for some customers. We are investigating.

This maintenance window is to:

• Test database replication failover
• Update Kubernetes ingress to remove TLS 1.0 support

There will be sporatic outages during this period. Thanks for your patience while we perform these changes and improvements.

[Update 2019-01-31 11:38 PST]
The outage is resolved and CW Kanban instances are available. Some ConnectWise data sync jobs are delayed but are being reloaded now.

==============================

[Update 2019-01-31 10:55 PST]
The hosting provider has addressed a DDoS attack that caused the outage. We are now working to bring back connectivity in our application cluster.

==============================

CW Kanban is unavailable due to networking issues with our hosting provider. We are investigating and working with the provider to bring the service back online.

[Update 2019-01-28 20:25 PST]
Maintenance is complete.
==============================

We are performing maintenance on CWKanban infrastructure on Monday evening Pacific Standard Time, beginning approximately 7:00 PM. Kanban boards may be unavailable during the work.

We are making cluster management services highly-available in order to improve availability in the future.

CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

[Update 2018-10-31 10:29 PST]

The server is back online and affect sites are available.

==============================

CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

[Update 2018-10-24 07:50 PST]

The server is back online and affect sites are available. We will be migrating sites to a different server soon.

==============================

[Update 2018-10-24 06:20 PST]

We're unable to bring the server online remotely so have asked our hosting provider to power cycle it.

==============================

CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

[Update 2018-10-18 8:30 PST]

CW Kanban service is restored for all customers. We will investigate and address the causes leading to the outage on this server.

==============================

CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

[Update 2018-10-17 15:40 PST]

CW Kanban service is restored for all customers. We will investigate and address the causes leading to the outage on this server.

==============================

CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

We have resolved the issue, all Kanban instances are available. We apologize for any inconvenience, and if you have any questions please feel free to make a support ticket.

Thank you for your patience with this matter.

CW Kanban is unavailable for some customers. We are working to bring it back online for affected customers.

CW Kanban is unavailable for some customers. We are working to bring it back online for all customers.

The datacenter where Kanban is hosted is having networking issues. All Kanban instances are unavailable.

The datacenter where Kanban is hosted is having power and networking issues. All Kanban instances are unavailable.

[Update 2018-02-13 22:00 PST]
The storage on our Kubernetes node has been upgraded and all instances are available again.
==============================

We are migrating Kanban instances to a new host. Service will be unavailable while this is taking place.

[Update 2018-01-08 20:00 PST]
The Linux kernels on the service guest and host are up-to-date. We will continue to monitor the system for any further issues.
==============================
[Update 2018-01-08 19:20 PST]
We are going to update the kernel packages on the guest and host. There will be some periods of unavailability in the next hour.
==============================
[Update 2018-01-08 19:11 PST]
Customer instances have started. The issue is resolved.
==============================
[Update 2018-01-08 19:00 PST]
The service guest had suffered a kernel panic and has been restarted. We are starting to bring up customer instances.
==============================

[Update 2017-12-05 12:25 PST]

We are restoring instance data to the latest data available on the old host, which was recovered earlier today. Kanban will be unavailable for a few minutes while the restore takes place.

At this time we do not plan to migrate back to the original host.
==============================
[Update 2017-12-05 00:47 PST]
We have brought up Kanban instances at a new hosting provider. DNS is updating now. To access your instance before DNS is updated, set a hosts file entry for 104.37.168.56.

We used backup data from morning of Sunday, December 3. If we regain access to our old host soon, we will update instances with latest data from the old host.

The old hosting provider reports the outage was due to hardware failure.
==============================
[Update 2017-12-04 19:42 PST]

We continue to await information from our hosting provider. We are building new infrastructure at a new hosting provider.
==============================
[2017-12-04 17:18 PST]

CW Kanban is unavailable due to an outage from our hosting provider. We are awaiting an update from our hosting provider.