CW Kanban unavailable 2019-12-03 (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2019-12-03 16:35 PST]
    We identified the networking issue in our infrastructure and applied a fix. All Kanban boards are available again.

    ==============================

    Kanban boards are unavailable due to a DNS problem in our infrastructure. We're investigating and will post updates here.

  • Date - 2019-12-03 15:58 - 2019-12-03 16:33
  • Last Updated - 2019-12-03 16:35
Embedded Kanban unavailable 2019-08-08 (Resolved)
  • Priority - High
  • Affecting Other - Embedded Kanban in ConnectWise Manage
  • Update 12:33 AM PST: The fix has been applied. The embedded kanban service has been restored.

    Update 11:22 AM PST: We are currently deploying the fix to all instances and expect this issue to be resolved shortly.

    Accessing embedded CW Kanban tabs in ConnectWise is unvailable for all customers. We believe this to be a change to the ConnectWise API and are currently working on a solution. Current ETA of fix is under a few hours.

  • Date - 2019-08-08 09:40 - 2019-08-08 12:32
  • Last Updated - 2019-08-08 12:34
Scheduled CW Kanban maintenance 2019-07-19 (Resolved)
  • Priority - Medium
  • Affecting Other - CW Kanban
  • We'll be performing server maintenance on 2019-07-19 between 5 PM and 8 PM PST. CW Kanban might be unavailable for periods during this window.

  • Date - 2019-07-19 17:00 - 2019-07-18 20:00
  • Last Updated - 2019-07-22 09:17
CW Kanban unavailable 2019-04-18 (Resolved)
  • Priority - Critical
  • Affecting Other - CWKanban
  • [Update 2019-04-18 09:30 PST]
    All sites are available. The cause appears to have been a network partition. All workloads have been moved off the unreachable host.
    ==============================

    CW Kanban is unavailable for some customers. We are investigating.

  • Date - 2019-04-18 08:44 - 2019-04-18 09:29
  • Last Updated - 2019-04-18 09:29
Scheduled CW Kanban maintenance 2019-03-14 (Resolved)
  • Priority - Medium
  • Affecting Other - CWKanban
  • This maintenance window is to:

    • Test database replication failover
    • Update Kubernetes ingress to remove TLS 1.0 support

    There will be sporatic outages during this period. Thanks for your patience while we perform these changes and improvements.

     

  • Date - 2019-03-14 17:00 - 2019-03-14 19:00
  • Last Updated - 2019-03-18 10:40
CW Kanban unavailable 2019-01-31 (Resolved)
  • Priority - Critical
  • Affecting Other - CWKanban
  • [Update 2019-01-31 11:38 PST]
    The outage is resolved and CW Kanban instances are available. Some ConnectWise data sync jobs are delayed but are being reloaded now.

    ==============================

    [Update 2019-01-31 10:55 PST]
    The hosting provider has addressed a DDoS attack that caused the outage. We are now working to bring back connectivity in our application cluster.

    ==============================

    CW Kanban is unavailable due to networking issues with our hosting provider. We are investigating and working with the provider to bring the service back online.

  • Date - 2019-01-31 10:12 - 2019-01-31 11:15
  • Last Updated - 2019-01-31 11:39
Scheduled CW Kanban maintenance 2019-01-28 (Resolved)
  • Priority - High
  • Affecting Other - CWKanban
  • [Update 2019-01-28 20:25 PST]
    Maintenance is complete.
    ==============================

    We are performing maintenance on CWKanban infrastructure on Monday evening Pacific Standard Time, beginning approximately 7:00 PM. Kanban boards may be unavailable during the work.

    We are making cluster management services highly-available in order to improve availability in the future.

  • Date - 2019-01-28 19:00 - 2019-01-28 20:20
  • Last Updated - 2019-01-28 20:24
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

  • Date - 2018-11-05 11:30 - 2018-11-05 11:50
  • Last Updated - 2018-11-05 12:29
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

  • Date - 2018-11-01 17:50 - 2018-11-01 21:40
  • Last Updated - 2018-11-02 09:20
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2018-10-31 10:29 PST]

    The server is back online and affect sites are available.

    ==============================

    CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

  • Date - 2018-10-31 09:10 - 2018-10-31 10:29
  • Last Updated - 2018-10-31 10:30
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2018-10-24 07:50 PST]

    The server is back online and affect sites are available. We will be migrating sites to a different server soon.

    ==============================

    [Update 2018-10-24 06:20 PST]

    We're unable to bring the server online remotely so have asked our hosting provider to power cycle it.

    ==============================

    CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

  • Date - 2018-10-24 04:45 - 2018-10-24 07:50
  • Last Updated - 2018-10-24 08:51
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2018-10-18 8:30 PST]

    CW Kanban service is restored for all customers. We will investigate and address the causes leading to the outage on this server.

    ==============================

    CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

  • Date - 2018-10-18 07:00 - 2018-10-18 08:30
  • Last Updated - 2018-10-18 08:35
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2018-10-17 15:40 PST]

    CW Kanban service is restored for all customers. We will investigate and address the causes leading to the outage on this server.

    ==============================

    CW Kanban is unavailable for some customers. We are investigating and bringing the affected server back online.

     

  • Date - 2018-10-17 14:55 - 2018-10-17 15:30
  • Last Updated - 2018-10-17 15:43
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • We have resolved the issue, all Kanban instances are available. We apologize for any inconvenience, and if you have any questions please feel free to make a support ticket.

    Thank you for your patience with this matter.

  • Date - 2018-09-19 10:24 - 2018-09-19 12:25
  • Last Updated - 2018-09-19 12:26
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • CW Kanban is unavailable for some customers. We are working to bring it back online for affected customers.

  • Date - 2018-07-20 10:50 - 2018-07-20 11:05
  • Last Updated - 2018-07-20 11:04
CW Kanban unavailable for some customers (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • CW Kanban is unavailable for some customers. We are working to bring it back online for all customers.

  • Date - 2018-07-09 06:30 - 2018-07-09 07:00
  • Last Updated - 2018-07-09 07:03
CW Kanban unavailable 2018-03-17 (Resolved)
  • Priority - Critical
  • Affecting Other - Kanban
  • The datacenter where Kanban is hosted is having networking issues. All Kanban instances are unavailable.

  • Date - 2018-03-17 13:30 - 2018-03-17 14:45
  • Last Updated - 2018-03-19 09:07
CW Kanban unavailable 2018-02-18 (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • The datacenter where Kanban is hosted is having power and networking issues. All Kanban instances are unavailable.

  • Date - 2018-02-18 09:00 - 2018-02-18 14:30
  • Last Updated - 2018-02-18 15:56
Scheduled CW Kanban maintenance 2018-02-13 (Resolved)
  • Priority - Medium
  • Affecting Other - CW Kanban
  • [Update 2018-02-13 22:00 PST]
    The storage on our Kubernetes node has been upgraded and all instances are available again.
    ==============================

    February 13, 2018 between 8 and 11 PM PST we will be increasing storage available to one of our Kubernetes nodes. Kanban for some customers will be unavailable during the update.

  • Date - 2018-02-13 11:09 - 2018-02-13 22:00
  • Last Updated - 2018-02-13 21:59
Scheduled CW Kanban maintenance 2018-02-08 (Resolved)
  • Priority - Medium
  • Affecting Other - CW Kanban
  • We are migrating Kanban instances to a new host. Service will be unavailable while this is taking place.

  • Date - 2018-02-08 21:08
  • Last Updated - 2018-02-08 22:58
Scheduled CW Kanban maintenance 2018-01-11 (Resolved)
  • Priority - Medium
  • Affecting Other - CW Kanban
  • [Update 2018-01-12 00:35 PST]
    The application and and infrastructure updates are complete.
    ==============================
    [Update 2018-01-11 21:50 PST]
    Unexpected issues have impacted the application upgrade. No customer instances have been updated, and all continue to be available. However we will extend the maintenance window until midnight PST.
    ==============================

    On January 11, 2018 between 8 and 10 PM PST, we will be performing application and infrastructure updates. All customers will have periods of unavailability during this window.

    The application updates relate to improvements in synchronizing data with ConnectWise, and the infrastructure updates relate to Linux kernel updates to deal with Meltdown/Spectre.

    We will update this record when the maintenance is complete.
     

  • Date - 2018-01-11 20:00 - 2018-01-12 00:35
  • Last Updated - 2018-01-12 00:37
CW Kanban unavailable 2018-01-08 (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2018-01-08 20:00 PST]
    The Linux kernels on the service guest and host are up-to-date. We will continue to monitor the system for any further issues.
    ==============================
    [Update 2018-01-08 19:20 PST]
    We are going to update the kernel packages on the guest and host. There will be some periods of unavailability in the next hour.
    ==============================
    [Update 2018-01-08 19:11 PST]
    Customer instances have started. The issue is resolved.
    ==============================
    [Update 2018-01-08 19:00 PST]
    The service guest had suffered a kernel panic and has been restarted. We are starting to bring up customer instances.
    ==============================

    CW Kanban is unavailable for all customers. We are investigating the cause. The hypervisor is available and appears to be operating normally, but CPU use on the service guest is elevated.

  • Date - 2018-01-08 18:26
  • Last Updated - 2018-01-08 20:00
CW Kanban unavailable 2017-12-04 (Resolved)
  • Priority - Critical
  • Affecting Other - CW Kanban
  • [Update 2017-12-05 12:25 PST]

    We are restoring instance data to the latest data available on the old host, which was recovered earlier today. Kanban will be unavailable for a few minutes while the restore takes place.

    At this time we do not plan to migrate back to the original host.
    ==============================
    [Update 2017-12-05 00:47 PST]
    We have brought up Kanban instances at a new hosting provider. DNS is updating now. To access your instance before DNS is updated, set a hosts file entry for 104.37.168.56.

    We used backup data from morning of Sunday, December 3. If we regain access to our old host soon, we will update instances with latest data from the old host.

    The old hosting provider reports the outage was due to hardware failure.
    ==============================
    [Update 2017-12-04 19:42 PST]

    We continue to await information from our hosting provider. We are building new infrastructure at a new hosting provider.
    ==============================
    [2017-12-04 17:18 PST]

    CW Kanban is unavailable due to an outage from our hosting provider. We are awaiting an update from our hosting provider.

  • Date - 2017-12-04 16:57
  • Last Updated - 2018-01-08 19:15