SWAN Member Library Staff:
The Symphony/WorkFlows server is back online and available for logins. Please login normally at your earliest convenience.
We continue to troubleshoot issues relating to SIP and the system load, so we may experience additional outages our slowness during the afternoon.
We'll send additional updates via SWANcom and support site post<https://support.swanlibraries.net/news/2018-12/66379> as information becomes available.
Thank you.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: Steven Schlewitt
Sent: Tuesday, January 15, 2019 10:16 AM
To: 'swancom(a)list.swanlibraries.net' <swancom(a)list.swanlibraries.net>; 'swantech(a)list.swanlibraries.net' <swantech(a)list.swanlibraries.net>; swancirc(a)list.swanlibraries.net
Subject: WorkFlows Continued Outage on 1/15, 10:15am - *Please use Offline*
SWAN Member Library Staff:
We're currently experiencing an outage of the Symphony server. Please proceed to use WorkFlows in Offline Mode<https://support.swanlibraries.net/documentation/64828> as we continue to troubleshoot.
We sincerely apologize for the inconvenience and continue to work to find a resolution to these ongoing issues. We appreciate your continued patience.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
SWAN Member Library Staff:
We're currently experiencing an outage of the Symphony server. Please proceed to use WorkFlows in Offline Mode<https://support.swanlibraries.net/documentation/64828> as we continue to troubleshoot.
We sincerely apologize for the inconvenience and continue to work to find a resolution to these ongoing issues. We appreciate your continued patience.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
SWAN Member Library Staff--
I wanted to update everyone on the status of the new server supporting Symphony. The SWAN IT has been updating the support website throughout the day, so make sure to keep this URL open and refreshed throughout the day tomorrow.
Server Migration Update:
https://support.swanlibraries.net/news/2018-12/66379
The good news
This server is well-provisioned and responding in the way we expected. Libraries will be happy with the difference once we have stabilized our configuration.
We have positioned ourselves for Symphony to be stable for the rest of the evening. *Start your Tuesday as normal, and watch for updates via SWANcom and the SWAN support website.*
The bad news
Throughout the day today, SWAN's Symphony server experienced huge spikes in server load which caused WorkFlows connections to drop. We have identified a probable cause related to the SIP2 service.
The SIP2 service SWAN provides libraries is well over 100 connections. These are connections to self-checks, automated material handlers, PC Reservation, MyPC, etc. We regret that these systems have been affected by the server migration. We are examining the SIP configuration library by library, and have randomized the reactivation of SIP connections as we work through them.
Acquisitions Symphony Reports remain disabled today, but we are looking towards allowing these reports to be run after we regroup in the morning.
Support staff will continue to monitor the system throughout the evening.
Thank you for your patience and we appreciate your support.
Aaron Skog
Executive Director
aaron(a)swanlibraries.net<mailto:aaron@swanlibraries.net> * (630)326-7022
SWAN Library Services
800 Quail Ridge Drive, Westmont, IL 60559
844-792-6542
swanlibraries.net<https://www.swanlibraries.net/> * catalog.swanlibraries.net<https://catalog.swanlibraries.net/>
SWAN Member Library Staff:
We've again disabled SIP2 connectivity as we continue to troubleshoot. If possible, please leave your SIP devices connected to the server (do not power them off), and we may need to review that traffic in our logs.
Thank you.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: Steven Schlewitt
Sent: Monday, January 14, 2019 1:07 PM
To: swancom(a)list.swanlibraries.net; swantech(a)list.swanlibraries.net; swancirc(a)list.swanlibraries.net
Subject: *Monitoring* WorkFlows Continued Outage on 1/14, 12pm - Please Sign In Now
Importance: High
SWAN Member Library Staff:
The Symphony server is again stable and we're running additional monitoring processes to further troubleshoot.
Please feel free to login to WorkFlows again now. We've also reenabled SIP connectivity. We may see additional spikes this afternoon, so please be prepared to use Offline Mode<https://support.swanlibraries.net/documentation/64828> should you get disconnected abruptly.
Thank you for your continued patience as we work through the issue.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: Swancirc <swancirc-bounces(a)list.swanlibraries.net<mailto:swancirc-bounces@list.swanlibraries.net>> On Behalf Of Steven Schlewitt
Sent: Monday, January 14, 2019 12:07 PM
To: swancom(a)list.swanlibraries.net<mailto:swancom@list.swanlibraries.net>; swantech(a)list.swanlibraries.net<mailto:swantech@list.swanlibraries.net>; swancirc(a)list.swanlibraries.net<mailto:swancirc@list.swanlibraries.net>
Subject: [Swancirc] WorkFlows Continued Outage on 1/14, 12pm - *Please use Offline*
Importance: High
SWAN Member Library Staff:
We're currently seeing another load spike on the server. Please proceed to use WorkFlows in Offline Mode<https://support.swanlibraries.net/documentation/64828> as we continue to troubleshoot.
We sincerely apologize for the inconvenience and are working with SirsiDynix engineers to find a resolution.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
_______________________________________________
Swancirc mailing list
Swancirc(a)list.swanlibraries.net<mailto:Swancirc@list.swanlibraries.net>
http://list.railslibraries.info/cgi-bin/mailman/listinfo/swancirc
SWAN Member Library Staff:
The Symphony server is again stable and we're running additional monitoring processes to further troubleshoot.
Please feel free to login to WorkFlows again now. We've also reenabled SIP connectivity. We may see additional spikes this afternoon, so please be prepared to use Offline Mode<https://support.swanlibraries.net/documentation/64828> should you get disconnected abruptly.
Thank you for your continued patience as we work through the issue.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: Swancirc <swancirc-bounces(a)list.swanlibraries.net> On Behalf Of Steven Schlewitt
Sent: Monday, January 14, 2019 12:07 PM
To: swancom(a)list.swanlibraries.net; swantech(a)list.swanlibraries.net; swancirc(a)list.swanlibraries.net
Subject: [Swancirc] WorkFlows Continued Outage on 1/14, 12pm - *Please use Offline*
Importance: High
SWAN Member Library Staff:
We're currently seeing another load spike on the server. Please proceed to use WorkFlows in Offline Mode<https://support.swanlibraries.net/documentation/64828> as we continue to troubleshoot.
We sincerely apologize for the inconvenience and are working with SirsiDynix engineers to find a resolution.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
_______________________________________________
Swancirc mailing list
Swancirc(a)list.swanlibraries.net<mailto:Swancirc@list.swanlibraries.net>
http://list.railslibraries.info/cgi-bin/mailman/listinfo/swancirc
SWAN Member Library Staff:
We're currently seeing another load spike on the server. Please proceed to use WorkFlows in Offline Mode<https://support.swanlibraries.net/documentation/64828> as we continue to troubleshoot.
We sincerely apologize for the inconvenience and are working with SirsiDynix engineers to find a resolution.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
Just to further clarify - the Symphony/WorkFlows server is now accessible. Please login to WorkFlows normally.
SIP2 connectivity (for AMH, self-check, PC reservation) is currently down, but we'll be opening the connection again shortly.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: Steven Schlewitt
Sent: Monday, January 14, 2019 9:53 AM
To: swancom(a)list.swanlibraries.net; swancom(a)list.swanlibraries.net; swantech(a)list.swanlibraries.net
Subject: *Resolved* SWAN Server Migration Issues on 1/14 *Please Use Offline*
SWAN Member Library Staff:
We believe to have pinpointed the issue that caused our early morning load spike. We've reopened connections to WorkFlows, but continue to troubleshoot issues with the SIP2 service.
Please login to WorkFlows as usual and await further updates regarding the SIP2 service.
Thank you for your patience.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: SWANcom <swancom-bounces(a)list.swanlibraries.net<mailto:swancom-bounces@list.swanlibraries.net>> On Behalf Of Rudy Host
Sent: Monday, January 14, 2019 8:47 AM
To: swancom(a)list.swanlibraries.net<mailto:swancom@list.swanlibraries.net>
Subject: [SWANcom] SWAN Server Migration Issues *Please Use Offline*
SWAN Member Library Staff:
We are currently experiencing some high load issues with the Symphony server. Please use offline mode while we investigate the problem. We will send an update when we have more information and when it is safe to log in normally.
Thank you for your patience throughout the migration process.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
SWAN Member Library Staff:
We believe to have pinpointed the issue that caused our early morning load spike. We've reopened connections to WorkFlows, but continue to troubleshoot issues with the SIP2 service.
Please login to WorkFlows as usual and await further updates regarding the SIP2 service.
Thank you for your patience.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
From: SWANcom <swancom-bounces(a)list.swanlibraries.net> On Behalf Of Rudy Host
Sent: Monday, January 14, 2019 8:47 AM
To: swancom(a)list.swanlibraries.net
Subject: [SWANcom] SWAN Server Migration Issues *Please Use Offline*
SWAN Member Library Staff:
We are currently experiencing some high load issues with the Symphony server. Please use offline mode while we investigate the problem. We will send an update when we have more information and when it is safe to log in normally.
Thank you for your patience throughout the migration process.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
SWAN Member Library Staff:
We are currently experiencing some high load issues with the Symphony server. Please use offline mode while we investigate the problem. We will send an update when we have more information and when it is safe to log in normally.
Thank you for your patience throughout the migration process.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/
SWAN Member Library Staff:
We've successfully completed our server infrastructure migration to the cloud! The new SWAN Symphony server in the Microsoft Azure Cloud has finished the regularly scheduled overnight maintenance reports and is now available for use.
A few things to note:
* Please login to WorkFlows and perform all functions as usual. No further downtime is expected today. Additionally, we will not need to use Offline this morning.
* Acquisitions departments - we'll be performing some additional testing later this morning before reenabling acquisitions reports. Please watch for an update later this morning.
* Our report schedule is currently on-track, with most reports running as scheduled.
* All regularly scheduled reports will run as expected today. This includes autorenewals, patron notices, and clean holds shelf.
* BLUECloud Analytics statistics may be delayed by one day. We expect this will be caught up tomorrow (Tuesday) morning.
* All Enterprise, payment, and authentication services are operating as expected, with the exception of item availability in the search display due to ongoing troubleshooting.
* The new server uses the hostname api.in.swanlibraries.net (same as our previous server). As long as your WorkFlows client is pointing to this hostname, you'll login to the server as usual.
* No changes are needed to SIP2 devices or software referring to the current hostname.
We'll send additional SWANcoms and news posts if any substantial issues arise, so please continue to watch throughout the day. Additionally, if you encounter any issues or oddities, please send them to help(a)swanlibraries.net<mailto:help@swanlibraries.net> so that we can investigate further.
Thank you for your patience throughout the migration process.
--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help(a)swanlibraries.net<mailto:help@swanlibraries.net>
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/