2/26/19-3/10/19 Service Disruption - Resolved
03/10/19 10:30 PM - We have been monitoring our connection success rate for the past 3 days. Everyday our connection rate is within 0.5% fluctuation from our connection rate before the incident. We feel good to call the issue being resolved. Internally we will run post analysis on this issue to see how we can improve response time and minimize issue impact. Thanks!
03/08/19 12:00 AM - Our provider did server load balancing about 24 hours ago. From connection metrics we see connection rate went up 2%-3%, which is very close to what we had before this service issue. We still higher volume of connection reports from customers. We are monitoring.
03/07/19 1:00 AM - We have been working with our service provider on this. The server capacity is enough but there is load balancing issue on some servers. Our service provider is going to adjust the load balance to make sure no server has very high requests. We look forward to this change to get back to normal service status. Sorry for the wait!
03/06/19 2:00 PM - Starting Monday we saw increased instances of connectivity issue. From our metrics we saw 2%-3% drop for connection rate. The failure is not as much as last week but frequent enough to keep this issue open. Most of the failure symptom was connection timed out or video froze few seconds after connection. The issue affected iOS more than Android. Again, this issue only affects a group of devices, not all devices.
We are actively working with our provider on this. The servers capacity is fine but some other metrics are not normal. We are investigating...
03/03/19 12:20 PM - In the past two days we see our connection metric back to previous level. We are close to call the service disruption to be resolved.
03/01/19 12:30 PM - More customers are confirming their cameras are back. Our metric also showed the connection rate went up 10% comparing yesterday. We have the last few base points to catch up. There are still a small percentage of devices having connection issue, either can't connect or connection freezes right after connection. We are working on it.
03/01/19 12:50 AM - There is another cloud deployment happened around 2/28 11:30pm. From a few customer responses connection came back. We will keep monitoring in daytime.
02/28/19 8:15 PM - Confirmed the issue only affects a few groups of devices, not all devices. We have seen more iOS connection failure/timeout than Android. On Android, connection may be slow but eventually connection will succeed.
With the additional cloud servers added for the service, we saw about 10% improvement but we are still around 15% behind our regular connection success rate. It seems like server throughput is still an issue. We are working on another solution in next few hours.
02/28/19 12:36 PM - Due to additional reports coming in of continued remote connection failures, we've re-opened the investigation into our streaming service.
02/27/19 11:00 PM - We identified the issue and deployed a fix for it. We saw good number of devices were back. It is promising to see the issue being resolved.
02/26/19 - We're receiving reports from some people that they are having connection issues when attempting to access the live streams of their Wyze Cams. This is a high priority for us and we'll update here when we have more information.
1/17/19 Service Disruption - Resolved
We are aware of an issue causing cameras to not connect to the live-stream. We are actively investigating and hope to have an update as soon as possible. Thank you for your patience.
Update: 1/17/19 12:13 PM PT - No further updates at this time. Thank you for your patience.
Update: 1/17/19 1:03 PM PT - No further updates at this time. We are working with our engineers on this issue. We appreciate your patience.
Update: 1/17/19 2:08 PM PT - We are seeing improved connection across users but are still investigating the issue.
Update: 1/17/19 2:50 PM PT - We are seeing improvement and connectivity is improving. However, we are still investigating the root cause and aren't calling this resolved yet.
Update: 1/17/19 5:00 PM PT - We are seeing obvious improvements with the connection. A large number of devices are recovered. We are waiting for more data before we call the issue resolved.
Update: 1/18/19 10:00 AM PT - We expect the issue should be mostly resolved. If you are still experiencing issues, please contact support
11/24/18 - Service Disruption - Resolved
- Impacts only Wyze Cam v2 and Wyze Cam Pan. Wyze Cam v1 is not impacted.
Update: 11/29 1:00am PST - After few days of monitoring our metrics we are confident to call the issue has been resolved.
Root cause of the issue: Our service provider made an service upgrade which aimed to better support the US market. However during the deployment there was a bug introduced into the system which caused camera login failed for Wyze cameras. That caused connection outside home network failed since the device showed as offline for streaming. Around 11/26 midnight the issue was identified and fixed. However a device restart needs to happen for each camera for re-login. Given our customer size it took about 2+ days for most cameras to be back working.
Again, we are very sorry for the service disruption! Thanks for your patience and continuous support with us! We have learned a lot from the two service issues last week. That is definitely not our bar. We are determined to give our customers a much better experience. We sincerely hope we can use our actions to earn your trust again!
Update: 11/26 6:45PM PST - Customer feedback positive but are waiting for service data to come back to normal
After we posted our solution at 7:25am a lot of customers verified the problem was resolved with the 'device restart' solution. Our support team also confirmed the fix with our customers today. However we are not calling the issue resolved yet. Our connection metric is still lower than what it was before. We want to see the metric to be very close to the past.
Update: 11/26 2:30PM PST - Lingering service issues still occurring and are still being investigated.
Update: 11/26 7:25AM PST - Provider updated that the issue is now resolved. If your device is not working, you need to either power your camera (by unplugging it and plugged back in) or restart your camera by going to Camera Settings > Restart Device.
Update: 1:23PM PST - Temporary Solution.
We are still working on the issue. Before we are able to get to the bottom of it, we are offering a temporary solution.
You can use this weblink https://user.wyzecam.com/upgradedevices to upgrade your camera firmware.
Instruction when used on mobile phone ( desktop version is more straight forward).
Step 1 - Click this URL https://user.wyzecam.com/upgrade
Step 2 - Log-in with your Wyze account
Step 3 - If you see a list of your devices, then go to step 5. Alternatively, If you see a welcome screen like below, then click the red button on top right.
Step 4 - Click "User information" and then "Upgrade Devices".
Step 5 - Scroll the device list screen all the way to the right, click "upgrade" button.
Please note that this is a temporary solution. We are committed to find the root cause and implement a long-term solution. So, if you are not willing to use this temporary solution, we appreciate your patience while we working to fully resolve the issue.
For details about this temporary solution, please see below.
Our P2P (peer-to-peer) service provider uses global load balancing to optimize live stream connection reliability and speed.
After we launched Wyze Cam, we heard concerns from our community about traffic going outside of the United States due to the global load balancing. We then implemented “Server Restriction” to limit traffic to only US-based servers. Although we don’t think the server restriction is the root cause, it is related to this issue.
By removing the server restriction, a user should be able to view the live stream outside of their local network.
What does this mean?
1. For the Live Stream, the P2P servers are still North America Only. So the live streaming video still stays inside North America.
The change is that in order to make device registration successful, a global list of servers (not only US-based) will be pinged, and the fastest server will respond for device registration. Before making this change, it would only use US servers.
2. For Alert Videos there is NO CHANGE. Your alert video will still go to US-based AWS servers (Owned by Wyze) with the same encryption and security level.
Update: 11:51AM PST - We are still working on the issue.
Update: 10:13AM PST - We are still working on the issue.
Update: 8:00AM PST - We are still working on the issue.
Update: 5:21AM - We got a local repro minutes ago with extra log info. We are comparing the local log and server log to search for root cause.
Starting 11/24, we noticed that some cameras cannot view the live stream outside of the local network (error 90 or 42). This is close to 11/19 issue discussed below, but the root cause has not been determined. We are working on fixing it, and will provide updates as we progress. We're sorry for the inconvenience.
11/19/18 - Service Disruption - Resolved
Update: 6:13PM PST - All devices should be back online. If yours hasn't, please try to Power Cycle them. (Unplug and plug back in.)
Update: 3:13PM PST - Most devices should be back online now. If yours hasn't, please give them a little more time or try to Power Cycle them. (Unplug and plug back in.)
Update: 2:58PM PST - Most cameras should be back online. Any that currently are not should be connecting soon.
Update: 1:38PM PST - Remnants of the issue remain, some users may still be experiencing it. Team is investigating.
Update: 1:19PM PST - Issue appears resolved - the team will be observing for any further issues.
Service Disruption 8:30am PST:
Some cameras may be experiencing issues connecting to our servers. This results in the camera flashing blue and may give you Error Code 90 within the app. Our engineering team is working on it as quickly as possible; it is a top priority this morning. We apologize for the inconvenience and hope to have the issue resolved quickly.
- The issue appears to mostly be affecting cameras that may have recently been disconnected from the servers. If the camera is already connected to the network, it should be fine. We recommend not resetting the connected cameras because they may lose the connection until this issue is resolved on our servers. If a camera loses connection, it may not be able to connect back for the time being until this problem is resolved.
- At this time, any new camera setups may not be able to complete.
- The issue does not appear to be affecting Event Recordings.
Resolved iOS issues
V1.5.50 - App crashes at splash screen on some iOS versions (10.3.3, 9.5.3, 9.3.5)
- Resolved 9/20. Please update app to iOS V1.5.51.
Resolved Android issues
V1.5.42 - App crashes at the splash screen with Android OS 5.0 or 5.1
- 9/20 update: Please update app to Android 1.5.44.