11/24/18 - Service Disruption
- Impacts only Wyze Cam v2 and Wyze Cam Pan. Wyze Cam v1 is not impacted.
Update: 11/29 1:00am PST - After few days of monitoring our metrics we are confident to call the issue has been resolved.
Root cause of the issue: Our service provider made an service upgrade which aimed to better support the US market. However during the deployment there was a bug introduced into the system which caused camera login failed for Wyze cameras. That caused connection outside home network failed since the device showed as offline for streaming. Around 11/26 midnight the issue was identified and fixed. However a device restart needs to happen for each camera for re-login. Given our customer size it took about 2+ days for most cameras to be back working.
Again, we are very sorry for the service disruption! Thanks for your patience and continuous support with us! We have learned a lot from the two service issues last week. That is definitely not our bar. We are determined to give our customers a much better experience. We sincerely hope we can use our actions to earn your trust again!
Update: 11/26 6:45PM PST - Customer feedback positive but are waiting for service data to come back to normal
After we posted our solution at 7:25am a lot of customers verified the problem was resolved with the 'device restart' solution. Our support team also confirmed the fix with our customers today. However we are not calling the issue resolved yet. Our connection metric is still lower than what it was before. We want to see the metric to be very close to the past.
Update: 11/26 2:30PM PST - Lingering service issues still occurring and are still being investigated.
Update: 11/26 7:25AM PST - Provider updated that the issue is now resolved. If your device is not working, you need to either power your camera (by unplugging it and plugged back in) or restart your camera by going to Camera Settings > Restart Device.
Update: 1:23PM PST - Temporary Solution.
We are still working on the issue. Before we are able to get to the bottom of it, we are offering a temporary solution.
You can use this weblink https://user.wyzecam.com/upgradedevices to upgrade your camera firmware.
Instruction when used on mobile phone ( desktop version is more straight forward).
Step 1 - Click this URL https://user.wyzecam.com/upgrade
Step 2 - Log-in with your Wyze account
Step 3 - If you see a list of your devices, then go to step 5. Alternatively, If you see a welcome screen like below, then click the red button on top right.
Step 4 - Click "User information" and then "Upgrade Devices".
Step 5 - Scroll the device list screen all the way to the right, click "upgrade" button.
Please note that this is a temporary solution. We are committed to find the root cause and implement a long-term solution. So, if you are not willing to use this temporary solution, we appreciate your patience while we working to fully resolve the issue.
For details about this temporary solution, please see below.
Our P2P (peer-to-peer) service provider uses global load balancing to optimize live stream connection reliability and speed.
After we launched Wyze Cam, we heard concerns from our community about traffic going outside of the United States due to the global load balancing. We then implemented “Server Restriction” to limit traffic to only US-based servers. Although we don’t think the server restriction is the root cause, it is related to this issue.
By removing the server restriction, a user should be able to view the live stream outside of their local network.
What does this mean?
1. For the Live Stream, the P2P servers are still North America Only. So the live streaming video still stays inside North America.
The change is that in order to make device registration successful, a global list of servers (not only US-based) will be pinged, and the fastest server will respond for device registration. Before making this change, it would only use US servers.
2. For Alert Videos there is NO CHANGE. Your alert video will still go to US-based AWS servers (Owned by Wyze) with the same encryption and security level.
Update: 11:51AM PST - We are still working on the issue.
Update: 10:13AM PST - We are still working on the issue.
Update: 8:00AM PST - We are still working on the issue.
Update: 5:21AM - We got a local repro minutes ago with extra log info. We are comparing the local log and server log to search for root cause.
Starting 11/24, we noticed that some cameras cannot view the live stream outside of the local network (error 90 or 42). This is close to 11/19 issue discussed below, but the root cause has not been determined. We are working on fixing it, and will provide updates as we progress. We're sorry for the inconvenience.
11/19/18 - Service Disruption
Update: 6:13PM PST - All devices should be back online. If yours hasn't, please try to Power Cycle them. (Unplug and plug back in.)
Update: 3:13PM PST - Most devices should be back online now. If yours hasn't, please give them a little more time or try to Power Cycle them. (Unplug and plug back in.)
Update: 2:58PM PST - Most cameras should be back online. Any that currently are not should be connecting soon.
Update: 1:38PM PST - Remnants of the issue remain, some users may still be experiencing it. Team is investigating.
Update: 1:19PM PST - Issue appears resolved - the team will be observing for any further issues.
Service Disruption 8:30am PST:
Some cameras may be experiencing issues connecting to our servers. This results in the camera flashing blue and may give you Error Code 90 within the app. Our engineering team is working on it as quickly as possible; it is a top priority this morning. We apologize for the inconvenience and hope to have the issue resolved quickly.
- The issue appears to mostly be affecting cameras that may have recently been disconnected from the servers. If the camera is already connected to the network, it should be fine. We recommend not resetting the connected cameras because they may lose the connection until this issue is resolved on our servers. If a camera loses connection, it may not be able to connect back for the time being until this problem is resolved.
- At this time, any new camera setups may not be able to complete.
- The issue does not appear to be affecting Event Recordings.
Resolved iOS issues
V1.5.50 - App crashes at splash screen on some iOS versions (10.3.3, 9.5.3, 9.3.5)
- Resolved 9/20. Please update app to iOS V1.5.51.
Resolved Android issues
V1.5.42 - App crashes at the splash screen with Android OS 5.0 or 5.1
- 9/20 update: Please update app to Android 1.5.44.